ESM - Intraindividual Variability Measures
Creators: name: Anvari, Farid
Publisher: name: Open Science Framework
Year: 2022
Subjects: subject: Personality and Social Contexts
subject: Social Psychology
subject: Social and Behavioral Sciences
subject: Psychology
subject: FOS: Psychology

description: This is a pre-registration for calculating variables and conducting correlation analyses between measures of intraindividual variability in positive and negative affect, personality, self-esteem, and variability in ratings for images. There is a .docx document ("EMA variability preregistration") that's attached which contains all details in navigable form (i.e., can use navigation pane to see the headings). I've copy and pasted from that document to here but the document has better formatting. Research Question The research questions examine (1) the extent to which Neuroticism and other scales designed to measure variability in mood and personality actually capture variability in mood and personality, respectively, as measured by daily ratings of mood and personality; and (2) whether intraindividual variability in repeated responses to self-report items is capturing some sort of systematic measurement error (e.g., a general tendency to vary in all things, or a general tendency to give more variable responses to the same questions when asked repeatedly). Sample and Methods/Procedure The target sample size is based on the data collection procedure is N = 700 for the baseline survey and N = 400 for the experience sampling study, or ecological momentary assessment study (EMA). The EMA sample would give two-sided correlation tests 95% power to detect r = .179, 90% power to detect r = .161, and 80% power to detect r = .14. It would also give equivalence tests (or the 90% CI) 98% power to reject (or exclude) r &gt;= |.2|, 92.4% power to reject r &gt;= |.17|, and 88% power to reject r &gt;= |.16|. There is a pre-registration for the data collection procedure and methods, including all measures that will be collected, on the Open Science Framework: https://osf.io/m9dz5. Hypotheses Hypothesis 1 If (i) the self-reported measures of variability in positive (PA) and negative affect (NA) and (ii) variability in multiple self-reports of positive and negative affect taken across time, both capture variability in the constructs of positive and negative affect, then there should be a positive correlation between: a) Negative Emotionality (i.e., Neuroticism) domain from BFI-2 and the adjusted index of variability in NA; b) Emotional Volatility facet from BFI-2 and the adjusted index of variability in NA; c) Volatility subscale from BFAS and the adjusted index of variability in NA; d) Affect Lability Short Scale (18-item) and the adjusted index of variability in NA; e) Affect Lability Short Scale (18-item) and the adjusted index of variability in PA. Hypothesis 2 If (i) the self-reported measures of variability in personality and (ii) variability in multiple self-reports of personality taken across time and contexts, both capture variability in personality, then there should be a positive correlation between: a) Self-Concept Clarity Scale and the adjusted index of variability in contextual personality; b) Self-Concept Clarity Scale and the adjusted index of variability in daily personality; c) Stability of Self Scale and the adjusted index of variability in contextual personality; d) Stability of Self Scale and the adjusted index of variability in daily personality; e) Self-Pluralism Scale and the adjusted index of variability in contextual personality; f) Self-Pluralism Scale and the adjusted index of variability in daily personality. Hypothesis 3 If (i) self-reported measures of variability in self-esteem and (ii) variability in multiple self-reports of self-esteem, both capture variability in self-esteem, then there should be a positive correlation between: Stability of Self Scale and the adjusted index of variability in daily self-esteem. Hypothesis 4 If variability in multiple self-reports taken across time and contexts capture some sort of systematic measurement error (e.g., a tendency to give variable responses to repeated self-reports; or a general tendency to be variable), then there should be a positive correlation between: a) The adjusted index of variability in NA and the adjusted index of variability in image ratings at baseline; b) The adjusted index of variability in PA and the adjusted index of variability in image ratings at baseline; c) The adjusted index of variability in contextual personality and the adjusted index of variability in image ratings at baseline; d) The adjusted index of variability in daily personality and the adjusted index of variability in image ratings at baseline; e) The adjusted index of variability in daily self-esteem and the adjusted index of variability in image ratings at baseline; f) The adjusted index of variability in NA and the adjusted index of variability in daily image ratings; g) The adjusted index of variability in PA and the adjusted index of variability in daily image ratings; h) The adjusted index of variability in contextual personality and the adjusted index of variability in daily image ratings; i) The adjusted index of variability in daily personality and the adjusted index of variability in daily image ratings; j) The adjusted index of variability in daily self-esteem and the adjusted index of variability in daily image ratings; In addition, there would also be a positive correlation between: k) The adjusted index of variability in NA and the adjusted index of variability in contextual personality; l) The adjusted index of variability in PA and the adjusted index of variability in contextual personality; m) The adjusted index of variability in NA and the adjusted index of variability in daily personality; n) The adjusted index of variability in PA and the adjusted index of variability in daily personality. o) The adjusted index of variability in NA and the adjusted index of variability in daily self-esteem; p) The adjusted index of variability in PA and the adjusted index of variability in daily self-esteem. q) The adjusted index of variability in daily self-esteem and the adjusted index of variability in contextual personality; r) The adjusted index of variability in daily self-esteem and the adjusted index of variability in daily personality. Analyses The variables to test each of the above hypotheses are described below. Each of the above hypotheses will be tested with two-sided, pairwise, Pearson’s correlation coefficient tests against a null hypothesis of zero. (Although the predictions are one-sided, two-sided tests can also falsify the predictions.) Therefore, for each hypothesis above, the corresponding measures listed below will be used. Items and Measures All items taken at the baseline survey and in the surveys during the mobile phone app study can be found in the preregistration of the data collection protocol (https://osf.io/m9dz5). Nonetheless, the items used for this particular study are as follows. Items in Baseline Survey Negative Emotionality domain from BFI-2 The items are rated on scale where 1=Disagree strongly, 2=Disagree a little, 3=Neutral; no opinion, 4=Agree a little, 5=Agree strongly. The items with ‘r’ next to the number are reverse scored and scores for the items on the domain are averaged. Negative Emotionality Anxiety 19. Can be tense. 34. Worries a lot. 4r. Is relaxed, handles stress well. 49r. Rarely feels anxious or afraid. Depression 39. Often feels sad. 54. Tends to feel depressed, blue. 9r. Stays optimistic after experiencing a setback. 24r. Feels secure, comfortable with self. Emotional Volatility 14. Is moody, has up and down mood swings. 59. Is temperamental, gets emotional easily. 29r. Is emotionally stable, not easily upset. 44r. Keeps their emotions under control. Emotional Volatility facet from BFI-2 (see above for items) Scores for the Emotional Volatility facet items are averaged after reverse scoring the relevant items. Volatility Subscale from BFAS Because a couple of the items (7 and 4) are virtually the same as the items on the BFI-2 (14 and 44, respectively), I included the BFAS items in the same block of questions as the BFI-2 on Qualtrics with slight adjustments to the wording on the items to fit with the sentence stem of the BFI-2. The wording I used for each item is presented in brackets next to the item on the below list. The items with the strikethrough are those that are already in the BFI-2 and the numbers in the brackets after the item reflect the position of the item in the BFI-2 as presented in Qualtrics BFI-2 question export tags. The items below are thus rated on the same 5-point scale as the BFI-2: 1=Disagree strongly, 2=Disagree a little, 3=Neither agree nor disagree, 4=Agree a little, 5=Agree strongly (numbers aren’t presented on the scales). Ratings on these items are averaged, after reverse scoring the items with (R) next to them. 1. Get angry easily. (66; Gets angry easily.) 2. Rarely get irritated. (R) (67; Rarely gets irritated.) 3. Get upset easily. (68; Gets upset easily.) 4. Keep my emotions under control. (R) (44; Keeps their emotions under control.) 5. Change my mood a lot. (69; Changes their mood a lot.) 6. Rarely lose my composure. (R) (70; Rarely loses their composure.) 7. Am a person whose moods go up and down easily. (14; Is moody, has up and down mood swings.) 8. Am not easily annoyed. (R) (71; Is not easily annoyed.) 9. Get easily agitated. (72; Gets easily upset.) 10. Can be stirred up easily. (73; Can be stirred up easily.) Affect Lability Short Scale 18-item Affect Lability Scale (Oliver &amp; Simons, 2004; see also Look et al., 2010): “How descriptive of you are the following statements?” from 1 = very undescriptive, to 4 = very descriptive. The total scale is summed for a total score on affect lability. 1. At times I feel just as relaxed as everyone else and then within minutes I become so nervous that I feel light-headed and dizzy. 2. There are times when I have very little energy and then just afterwards I have about the same energy level as most people. 3. One minute I can be feeling OK and then the next minute I’m tense, jittery, and nervous. 4. I frequently switch from being able to control my temper very well to not being able to control it very well at all. 5. Many times I feel nervous and tense and then I suddenly feel very sad and down. 6. Sometimes I go from feeling extremely anxious about something to feeling very down about it. 7. I shift back and forth from feeling perfectly calm to feeling uptight and nervous. 8. There are times when I feel perfectly calm one minute and then the next minute the least little thing makes me furious. 9. Frequently, I will be feeling OK but then I suddenly get so mad that I could hit something. 10. Sometimes I can think clearly and concentrate well one minute and then the next minute I have a great deal of difficulty concentrating and thinking clearly. 11. There are times when I am so mad that I can barely stop yelling and other times shortly afterwards when I wouldn’t think of yelling at all. 12. I switch back and forth between being extremely energetic and having so little energy that it’s a huge effort just to get where I am going. 13. There are times when I feel absolutely wonderful about myself but soon afterwards I often feel that I am just about the same as everyone else. 14. There are times when I’m so mad that my heart starts pounding and/or I start shaking and then shortly afterwards I feel quite relaxed. 15. I shift back and forth between being very unproductive and being just as productive as everyone else. 16. Sometimes I feel extremely energetic one minute and then the next minute I might have so little energy that I can barely do a thing. 17. There are times when I have more energy than usual and more than most people and then soon afterwards I have about the same energy level as everyone else. 18. At times I feel that I’m doing everything at a very slow pace but then soon afterwards I feel that I’m no more slowed down than anyone else. Self-Concept Clarity Scale 12-item Self-Concept Clarity Scale (Campbell et al. (1996): “How much do you agree or disagree with each of the following statements?” 1=Disagree strongly, 2=Disagree a little, 3=Neither agree nor disagree, 4=Agree a little, 5=Agree strongly (numbers aren’t presented on the scales). The items are presented in randomized order. The ratings are summed, after reverse scoring the items without a *. 1. My beliefs about myself often conflict with one another.* 2. On one day I might have one opinion of myself and on another day I might have a different opinion.* 3. I spend a lot of time wondering about what kind of person I really am.* 4. Sometimes I feel that I am not really the person that I appear to be.* 5. When I think about the kind of person I have been in the past, I’m not sure what I was really like.* 6. I seldom experience conflict between the different aspects of my personality. 7. Sometimes I think I know other people better than I know myself.* 8. My beliefs about myself seem to change very frequently.* 9. If I were asked to describe my personality, my description might end up being different from one day to another day.* 10. Even if I wanted to, I don’t think I would tell someone what I’m really like.* 11. In general, I have a clear sense of who I am and what I am. 12. It is often hard for me to make up my mind about things because I don’t really know what I want.* The items with * are typically reverse scored. However, for ease of interpretability of correlation coefficients, we reverse score the items that don’t have a *, since then higher scores on this scale would mean greater variability in personality. Stability of Self Scale 5-item Stability of Self Scale (Marsh, 1993): The original scale was on 5-point scale from never true, seldom true, sometimes true, often true, to almost always true. But we adapted it to be presented in the same block of questions as the Self-Concept Clarity Scale and therefore it has the same sentence stem and 5-point rating scales as above (numbers aren’t presented on the scales). The items are presented in randomized order within that block. The ratings are averaged after reverse scoring the items without a *. The numbers in the brackets to the right of each item reflects the position of that item in the Qualtrics export tag from the Self-Concept clarity block of items. 1. My opinion of myself tends to change a good deal instead of always remaining the same.* (13) 2. I find that on one day I have one opinion of myself and on another day I have a different opinion.* (2; see item 2 from self-concept clarity scale) 3. I often change from a very good opinion of myself to a very poor opinion of myself.* (14) 4. I have noticed that my ideas about myself seem to change very quickly.* (8; see item 8 from self-concept clarity scale) 5. I feel that nothing can change the opinion I currently hold of myself. (15) The items with * are typically reverse scored. However, for ease of interpretability of correlation coefficients, we reverse score the items that don’t have a *, since then higher scores on this scale would mean greater variability in personality. Note that for item 3 the word “often” was added because the original scale was about how often this statement was true but with the disagree-agree scale the item required some sort of frequency for the statement to make sense. Self-Pluralism Scale The Self-concept Clarity and Stability of Self scales may seem to tap a little more into self-esteem than into personality, assuming the two concepts are distinct. Therefore, we’ll also include the Brief Self-Pluralism Scale (SPS-10; McReynolds et al., 2000) with 10 items presented in random order. Participants are told to “Please indicate whether each of the following statements is true or false.”. False is coded as 0 and True is coded as 1 and the items are summed, after reverse scoring the items with a star, to produce a score with higher scores reflecting greater variability in personality: 1. I am the same kind of person in every way, day in and day out.* 2. People who know me well would say I act quite differently at different times. 3. Though I vary somewhat from time to time, in general I always feel much the same.* 4. People who know me well would say I’m pretty predictable.* 5. There have been times when I felt like a completely different person from what I was the day before. 6. I act and feel essentially the same way whether at home, at work, or with friends.* 7. I get along best when I act and feel like a totally different person at different times. 8. I sometimes have conflicts over whether to be one kind of person or a different kind. 9. I’m the same sort of person regardless of whom I’m with.* 10. People who know me would say that my behaviour changes from situation to situation. Adjusted Index of Variability in Contextual Personality Using the BFI-10 (Rammstedt &amp; John, 2007). “How much do you agree or disagree that each of the following statements describes your personality as a friend / family member / romantic partner / stranger?”. The following 10 items are then presented such that participants are asked to respond to them regarding how they are in the 4 different contexts. Each context is presented as part of a separate block of questions so that there are 4 blocks of questions. Among friends/ Among family/ When with a romantic partner/ Among strangers, I am someone who … 1. … is reserved 2. … is generally trusting 3. … tends to be lazy 4. … is relaxed, handles stress well 5. … has few artistic interests 6. … is outgoing, sociable 7. … tends to find fault with others 8. … does a thorough job 9. … gets nervous easily 10. … has an active imagination The items are presented in randomized order. Each of the items are rated on a 5-point scale: 1=Disagree strongly, 2= Disagree a little, 3=Neither agree nor disagree, 4=Agree a little, 5=Agree strongly (numbers aren’t presented on the scales). To calculate the index of variability in contextual personality, I’ll use the contextual personality measure following Baird et al. (2017): For each item, calculate the means and standard deviations across the different roles/contexts for each participant. You then regress each of these standard deviations for the different roles/contexts onto the corresponding mean and the square of the mean. You save the residuals from these regressions for each item, which can then be combined to create variability scores for each of the Big Five Traits and for overall variability. See Baird et al. (2018, p. 8). Therefore, for each item separately, assuming each item in the dataset has a similar suffix across the different roles/contexts: data$itemM &lt;- rowMeans(data[ , grep('Itemsuffix', colnames(data))], na.rm=TRUE) (need library(matrixStats) for rowSds() function) data$itemSD &lt;- rowSds(x = as.matrix(data[ , grep('itemsuffix', colnames(data))]), na.rm=TRUE) There are now 10 columns for each participant reflecting the cross-role/context means and standard deviations for each of the 10 personality items. For each item, I’ll regress the cross-role SD onto the mean and the square of the mean to obtain a residual. Model = lm(itemSD ~ itemM + itemM^2, data=data) data$RES_CP_item = resid(Model) RES_CP_item columns are now the corrected item-specific variability scores for contextual personality which can be combined to create variability scores for each of the Big Five Traits and an overall variability score. Can be combined by averaging them: Data$Variability_ContextPers = rowMeans (data[ , grep('RES_CP_', colnames(data))], na.rm=TRUE) Adjusted Index of Variability in Image Ratings at Baseline Attached is a zip folder called “ImageSets.zip”. This contains all of the images used in the baseline survey (described below) and also all of the images used in the daily image ratings (described further below). Ratings for ALL Grayscales The grayscale images were created by using a white rectangle and using a photo editor (https://www.befunky.com/create/photo-editor/) to add a black tinted filter on top of the image. And I varied the opacity of the tinted filter. Participants will be presented with each of the 14 images in random order and asked, “How white-black is this image?” on a 7-point scale ranging from 1 = “very white” to 7 = “very black”. I’ll calculate the adjusted variability in ratings for the grayscale images in the same way as done for the contextual personality measure. Using the grayscale ratings from the baseline survey (e.g., Grayscale1, Grayscale2 etc…): data$GrayM &lt;- rowMeans(data[ , grep(‘Grayscale’, colnames(data))], na.rm=TRUE) data$GraySD &lt;- rowMeans(data[ , grep('Grayscale', colnames(data))], na.rm=TRUE) model = lm(GraySD ~ GrayM + GrayM^2, data = data) data$RES_IM_Gray = resid(model) Ratings for ALL Blurry Images The blurred images were created by using a high quality image of two lady bugs and using a photo editor https://www.befunky.com/create/photo-editor/) to vary how blurry the image was. Participants will be presented with each of the 14 images in random order and asked, “How blurry-clear are the two ladybugs in this image?” on a 7-point scale ranging from 1 = “very blurry” to 7 = “very clear”. I’ll calculate the adjusted variability in ratings for the blurry images in the same way as done for the contextual personality measure. Using the blurry-clear ratings from the baseline survey (e.g., Blurry1, Blurry2 etc…): data$ BlurM &lt;- rowMeans(data[ , grep(‘Blurry’, colnames(data))], na.rm=TRUE) data$ BlurSD &lt;- rowMeans(data[ , grep('Blurry', colnames(data))], na.rm=TRUE) model = lm(BlurSD ~ BlurM + BlurM^2, data = data) data$RES_IM_Blur = resid(model) Ratings for ALL Vibrant Colour Images The colour saturation images were created by using a high quality image of a bird among spring flowers and using a photo editor https://www.befunky.com/create/photo-editor/) to vary how saturated the image was in colour. The variations ranged from 0 saturation (i.e., the standard colourful and vibrant image) with saturation reduced incrementally to create another 13 images until -100 saturation was reached (fully black and white photo). The images in Qualtrics are labelled with suffixes of 0 to 100, with 0 being fully vibrant (i.e., saturation is not reduced at all) and -100 being not at all vibrant (i.e., saturation set to -100). Participants will be presented with each of the 14 images in random order and asked, “How vibrant are the colours in this image?” on a 7-point scale ranging from 1 = “not at all” to 7 = “extremely”. I’ll calculate the adjusted variability in ratings for the vibrant colour images in the same way as done for the contextual personality measure. Using the ratings for how vibrant the colours of the images are from the baseline survey (e.g., Vibrant1, Vibrant2 etc…): data$VibM &lt;- rowMeans(data[ , grep(‘Vibrant’, colnames(data))], na.rm=TRUE) data$VibSD &lt;- rowMeans(data[ , grep(‘Vibrant’, colnames(data))], na.rm=TRUE) model = lm(VibSD ~ VibM + VibM^2, data = data) data$RES_IM_Vib = resid(model) Finally, assuming the three adjusted indices of variability in the image ratings at baseline are positively correlated, I’ll create a composite index of variability in the image ratings at baseline: data$Variability_Images = rowMeans(data[ , grep('RES_IM_', colnames(data))], na.rm=TRUE) Items in Experience Sampling Study Full details of the procedure and items in the experience sampling study are in the preregistered data collection protocol (https://osf.io/m9dz5). Adjusted Index of Variability in Positive and Negative Affect In the experience sampling study, we have measures of current state positive affect (positive, happy, proud, joyful, enthusiastic, and calm) and negative affect (negative, sad, ashamed, angry, anxious, and bored) taken up to 5 times per day for each participant, each item rated from 1 = not at all, to 5 = extremely. I’ll use the following code (or some adjustment to it depending on the data structure) to calculate the corrected intraindividual variability for each item (the example code is for the item “Happy”): data$HapM &lt;- rowMeans(data[ , grep('Happy', colnames(data))], na.action=na.exclude) data$HapSD &lt;- rowMeans(data[ , grep('Happy', colnames(data))], na.action=na.exclude) modelhap = lm(HapSD ~ HapM + HappyM^2, data = data) data$RES_PA_Hap = resid(modelhap) I’ll do this for each item on the positive affect scale and then we can create a composite score reflecting the variability in daily PA with the following code: data$ CorrectedSD_DailyPA = rowMeans(data[ , grep('RES_PA_', colnames(data))], na.rm=TRUE) And then I’ll do the same for the items on the negative affect scale to create an adjusted index of variability in NA. Adjusted Index of Variability in Daily Personality In the experience sampling study, the morning and evening surveys each have a 10-item measure of personality (i.e., personality is measured up to 2 times per day). The 10 items asks participants to rate how much they disagree-agree with the statement, ““Right now, I see myself as someone who ___.” with the blank filled in with one of the below 10 words/phrases presented in random order. Ratings are given on 5-point scales: 1=Disagree strongly, 2= Disagree a little, 3=Neither agree nor disagree, 4=Agree a little, 5=Agree strongly. The items are: 1. is reserved 2. is generally trusting 3. tends to be lazy 4. is relaxed, handles stress well 5. has few artistic interests 6. is outgoing, sociable 7. tends to find fault with others 8. does a thorough job 9. gets nervous easily 10. has an active imagination The adjusted index of daily personality ratings is calculated in the same as the adjusted index of variability in PA and NA above. For each participant, we calculate the mean of their ratings on each item, the standard deviation of their ratings on each item, and then we regress the mean and square of the mean for each item onto the item’s standard deviation to get the residual. The item’s residual is that participant’s adjusted variability score for the item. Then we average this across the 10 items for each participant to get an adjusted index of variability in daily personality ratings. data$itemM &lt;- rowMeans(data[ , grep('Itemsuffix', colnames(data))], na.rm=TRUE) data$itemSD &lt;- rowMeans(data[ , grep('Itemsuffix', colnames(data))], na.rm=TRUE) model = lm(itemSD ~ itemM + itemM^2, data = data) data$RES_DP_item = resid(model) # RES_DP_ stands for residual daily personality RES_DP_item columns are now the corrected item-specific variability scores for daily personality ratings which can be combined to create variability scores for each of the Big Five Traits and an overall variability score by averaging them: Data$Variability_DailyPers = rowMeans (data[ , grep('RES_DP_', colnames(data))], na.rm=TRUE) Adjusted Index of Variability in Daily Self-Esteem Self-esteem is measured with a single item once per day in the EMA (in the evening surveys). “Right now I have high self-esteem.”. Participants rate the statement on a 5-point scale: 1=Disagree strongly, 2= Disagree a little, 3=Neither agree nor disagree, 4=Agree a little, 5=Agree strongly. I’ll calculate the adjusted index of variability in daily self-esteem for this one item for each participant. Using the ratings for self-esteem taken each evening from the EMA phase: data$EstM &lt;- rowMeans(data[ , grep(‘Esteem’, colnames(data))], na.rm=TRUE) data$EstSD &lt;- rowMeans(data[ , grep(‘Esteem’, colnames(data))], na.rm=TRUE) model = lm(EstSD ~ EstM + VibrantM^2, data = data) data$Variability_SelfEst = resid(model) Adjusted Index of Variability in Daily Image Ratings Ratings for these are given on 7-point rating scales. Full details of the images can be found in the data collection pre-registration (https://osf.io/m9dz5). Once per day during the EMA study, participants will also be sent a Qualtrics study via the Prolific platform in which they’ll rate 3 sets of 2 images each on either how white-black the image is, how blurry-clear it is, or how vibrant the colours in the image are. One set of images contains grayscales that vary in how much tint covers a white background, one set contains an image of two ladybugs that vary in how blurry they are, and one set contains an image of a bird among flowers that varies in how vibrant the colours are. For each set of images for each participant, I’ll calculate an adjusted index of variability in the ratings for the images in that set. The example below is for grayscale images data$GrayDailyM &lt;- rowMeans(data[ , grep(‘Grayscale’, colnames(data))], na.rm=TRUE) data$GrayDailySD &lt;- rowMeans(data[ , grep('Grayscale', colnames(data))], na.rm=TRUE) model = lm(GrayDailySD ~ GrayDailyM + GrayDailyM^2, data = data) data$RES_IM_Gray = resid(model) Finally, assuming the three adjusted indices of variability in daily image ratings are positively correlated, I’ll create a composite index of variability in daily image ratings: data$Variability_DailyImages = rowMeans(data[ , grep('RES_IM_', colnames(data))], na.rm=TRUE) We're measuring within-person variability in (i) ratings for images at baseline (blurry-clear, white-black, vibrant colour), (ii) contextual personality measure at baseline, (iii) current personality measure taken across time in ESM (upto 14 measurement occasions per person), (iv) current PA and NA measures taken across time in ESM (upto 35 measurement occasions per person), and (v) current self-esteem measured across time in ESM (upto 7 measurement occasions per person). Variability in (i) should theoretically be unrelated to variability in any of the others. Therefore, the critical tests are whether (i) correlates with any one of (ii)-(v). This means that there are three tests for the hypothesis but we'll ignore that for now, since the strongest evidence would come from all of them being significant relationships. If we find nonsignificant correlations between variability in (i) and the variability in the other measures, (ii)-(v), and we rule out r &gt;= .16/.17/.2 for these correlations, then our conclusion would be that if the variability measures are capturing some sort of systematic measurement error, it's likely to be less than .16/etc.. We can further conclude that this should be further investigated. However, the results would suggest that at this stage we shouldn't be overly worried about systematic measurement error in the intraindividual variability measures. On the other hand, if we find a significant correlation between (i) and every one of (ii)-(v), then we'd conclude that the variability measures are capturing some sort of systematic measurement error, such as response styles or a general tendency to be more variable in all things. This would be a problem for those measures of within-person variability. Finally, if we find a significant correlation between (i) and any one of (ii)-(v) we'd draw a similar conclusion to the preceding paragraph but a bit more tentatively. If, on the other hand, the intraindividual variability in contextual and EMA personality measures capture systematic measurement error, then they should positively correlate with intraindividual variability in image ratings and those correlations should be stronger than the correlation between the intraindividual variability in contextual/EMA personality and self-reported variability in personality. Finally, look at how the volatility subscale of the BFI is linked to variability, and whether that link is larger than the anxiety and depression subscales. So can compare the correlation of the self-reported measures of variability in negative emotions with the corrected SD in daily NA, and compare these to the correlation of the depression and anxiety facets from BFI-II Neuroticism domain with the corrected SD in daily NA. Is the former stronger than the latter? If the intraindividual variability measures are capturing systematic measurement error in the form of response styles or global variation, then we expect: 1. Variability in daily PA and NA should be positively correlated with variability in contextual personality and variability in image ratings; 2. Variability in contextual personality, variability in daily personality, and variability in daily self-esteem should be positively correlated with variability in image ratings; 3. The relationships in (1) should be stronger than the correlation between variability in daily mood ratings and self-reported variability in mood; 4. The relationships in (2) should be stronger than the correlation of variability in contextual and daily personality with self-reported variability in personality; 5. Variability in daily PA and NA should show equivalence in relationship with self-reported variability in NA and self-reported variability in mood; and 6. Variability in contextual personality and variability in daily personality should show equivalence in relationship with self-reported variability in personality. If we observe that variability in daily PA and NA ratings are positively correlated with self-reported variability in mood then we can say that the different measures of variability in mood have some convergent validity. Similarly, if we observe that variability in contextual and daily personality ratings are positively correlated with self-reported variability in personality then we can say that the different measures of variability in personality have some convergent validity. If we observe that there is: 1. no correlation of variability in daily NA and PA with variability in image ratings AND 2. no correlation of variability in daily NA and PA with self-reported variability in mood then we conclude that the different measures of variability in mood don’t have convergent validity and that the measures of variability in daily mood ratings may not be capturing a type of response styles, or that if they are then that relationship is likely to be smaller than xxx (based on the CIs). Similarly, if we observe that there is: 1. no correlation of variability in contextual and daily personality ratings with variability in image ratings AND 2. no correlation of variability in contextual and daily personality ratings with self-reported variability in personality then we conclude that the different measures of variability in personality don’t have convergent validity and that the measures of variability in daily personality ratings may not be capturing a type of response styles, or that if they are then that relationship is likely to be smaller than xxx (based on the CIs). A note from Rich Lucas: And although I had forgotten that we did this study, we did in fact use a version of the color task you suggested, but we did it kind of haphazardly, and didn't think carefully about the colors we used. We also did not do it over time, we asked them about multiple colors (that varied from red to purple to blue and they had to rate how red versus blue they were) all at once. We also included some other measures, including rating how happy various faces were, the neutral objects questionnaire that we had in the other study, and then personality variability measure where people rate their personality in different roles. There was a consistent association among the measures, but it wasn't strong: r.faces r.colors r.neutral r.others r.personality r.faces 1.0000000 0.2487348 0.2662130 0.6710672 0.2093353 r.colors 0.2487348 1.0000000 0.1775679 0.6002129 0.0856562 r.neutral 0.2662130 0.1775679 1.0000000 0.7958955 0.3984736 r.others 0.6710672 0.6002129 0.7958955 1.0000000 0.3652643 r.personality 0.2093353 0.0856562 0.3984736 0.3652643 1.0000000 r.faces = residual of sd controlling for mean of rating of happiness; r.colors = residual of sd controlling for mean of colors; r.neutral = residual of neutral objects sd controlling for mean; r.others = average of the other three; r.personality = residual of personality variability controlling for mean. Thus, currently, it’s not at all clear whether the problem is with the self-reported variability in NA (i.e., Neuroticism) or with the measure of variability in NA (i.e., the corrected intraindividual variability in NA ratings). Kalokerinos et al. (2020) and others suggest the problem is with measures of Neuroticism. But this assumes that the truth criterion is the in the moment ratings. Baird et al. (2006, 2018) would suggest that there might also be a problem with the variability in the momentary ratings, since they might be capturing something much broader than only variability in that construct. If we find that corrected intraindividual variability in NA is correlated with corrected intraindividual variability in contextual personality and corrected intraindividual variability in image ratings, then we can argue that we don’t know whether the problem is with Neuroticism as a measure of emotional volatility or whether the problem is that intraindividual variability in NA ratings is capturing something much more global such that it is a general variability in people or response styles. Some evidence supports this already. The following Table is from an email from Rich Lucas based off of the data used in one of his papers (Baird et al., 2006, or 2018): pa_res = adjusted daily PA ratings; na_res = adjusted daily NA ratings, unadjPA = standard deviation in daily PA; unadjNA = standard deviation in daily NA; genn_i = neuroticism items from the contextual personality measures; neur_i = a measure of Neuroticism; neur_ii = another measure of Neuroticism. pa_res na_res unadjPA unadjNA genn_i pa_res 1.00000000 0.54614260 0.9962609 0.633356976 0.20500909 na_res 0.54614260 1.00000000 0.5445449 0.783666792 0.01478801 unadjPA 0.99626093 0.54454488 1.0000000 0.620855768 0.19890447 unadjNA 0.63335698 0.78366679 0.6208558 1.000000000 0.14278036 genn_i 0.20500909 0.01478801 0.1989045 0.142780362 1.00000000 neur_i 0.29292520 0.03817321 0.2835682 0.214554380 0.66520155 neur_ii 0.26572122 0.06294902 0.2513075 0.264493507 0.63570413 Although the standard deviation of daily NA ratings is correlated with N the adjusted intraindividual variability in daily NA ratings is NOT correlated with N. This replicates the findings of Kalokerinos et al. (2020) and others. (Note that both the standard deviation in daily PA and the adjusted intraindividual variability in daily PA are correlated with N.) However, Baird et al. (2018) found that the adjusted intraindividual variability in daily NA was correlated with intraindividual variability in contextual personality ratings, intraindividual variability in daily personality ratings, intraindividual variability in ratings of Neutral objects, and in daily weather ratings: This is preliminary evidence supporting my arguments above. But one could still argue that variability in daily NA ratings may be expected to be related to daily variability in weather ratings (and in weather actually), variability in satisfaction with neutral objects, variability in ratings of a close friend’s personality, and variability in daily ratings of own personality. In my study, the same expectation shouldn’t hold as strongly for variability in ratings for how white-black and how fuzzy-sharp images are. One could argue that variability might be a very global thing, so that variability in NA could be related to variability in everything else, because people may be globally variable. But such a finding would bring into question the claims of Kalokerinos et al. that N is a poor measure and future research would need to be done about whether variability in daily NA is part of global variability in people or whether it is measuring something like response styles. Exploratory Items and Measures In the evening surveys of the EMA study, participants will also be asked to report how much they experienced each of the emotions in the positive affect (positive, happy, proud, joyful, enthusiastic, and calm) and negative affect scales (negative, sad, ashamed, angry, anxious, and bored) during the whole day. Therefore, I can also calculate an index of adjusted variability in PA and NA using these items (see below). But the momentary ratings taken throughout the day, in real-time, don’t rely on memory, whereas these end-of-day reports on the whole day do. Therefore, the adjusted index of variability in end-of-day reports will be used for exploratory purposes only. Intraindividual variability in daily retrospective end-of-day reports Using the ratings for how people felt throughout the day taken each evening from the EMA phase, I’ll calculate an adjusted index of variability in daily retrospective PA reports and daily retrospective NA reports: data$HapDayM &lt;- rowMeans(data[ , grep('Happy_Day', colnames(data))], na.rm=TRUE) data$HapDaySD &lt;- rowMeans(data[ , grep('Happy_Day', colnames(data))], na.rm=TRUE) model = lm(HapDaySD ~ HapDayM + HapDayM^2, data = data) data$RES_PA_DayHap = resid(model) # we do this for each item on PA and then we can create a composite score reflecting the variability in daily PA in the following way data$Variability_DailyDayPA = rowMeans(data[ , grep('RES_PA_Day', colnames(data))], na.rm=TRUE) # and then do the same for the NA items.