# Statistical Evaluations of the Reproducibility and Reliability of 3-Tesla High Resolution Magnetization Transfer Brain Images: A Pilot Study on Healthy Subjects

Authors: Kelly H. Zou; Hongyan Du; Shawn Sidharthan; Lisa M. DeTora; Yunmei Chen; Ann B. Ragin; Robert R. Edelman; Ying Wu
Journal: International Journal of Biomedical Imaging
 (2010)

---

## Abstract

Magnetization transfer imaging (MT) may have considerable promise for early detection and monitoring of subtle brain changes before they are apparent on conventional magnetic resonance images. At 3 Tesla (T), MT affords higher resolution and increased tissue contrast associated with macromolecules. The reliability and reproducibility of a new high-resolution MT strategy were assessed in brain images acquired from 9 healthy subjects. Repeated measures were taken for 12 brain regions of interest (ROIs): genu, splenium, and the left and right hemispheres of the hippocampus, caudate, putamen, thalamus, and cerebral white matter. Spearman's correlation coefficient, coefficient of variation, and intraclass correlation coefficient (ICC) were computed. Multivariate mixed-effects regression models were used to fit the mean ROI values and to test the significance of the effects due to region, subject, observer, time, and manual repetition. A sensitivity analysis of various model specifications and the corresponding ICCs was conducted. Our statistical methods may be generalized to many similar evaluative studies of the reliability and reproducibility of various imaging modalities.


---

## Body

## 1. Introduction

Magnetization transfer (MT) imaging is a quantitative approach for detecting subtle or occult abnormalities in brain tissue. In previous studies, the Magnetization Transfer Ratio (MTR), an index of MT imaging, was sensitive to brain changes in patients with mild cognitive impairment, an Alzheimer’s disease prodrome [1, 2], to new lesions in patients with multiple sclerosis, [3] and to changes associated with progression in chronic neurological disorders [4]. The higher magnetic field strength afforded by 3T allows MT image resolution to be augmented compared with conventional MT acquisition at 1.5T [5–7]. We developed a high resolution MT technique to detect subtle changes in anatomically small, functionally eloquent brain structures. The increased field strength affords whole-brain coverage with considerably thinner slices, potentially reducing partial volume artifacts. However, even among healthy subjects, numerous factors may introduce variability in measures derived from magnetic resonance (MR) data, such as static field B0 signal dropout and RF nonuniformity. Measurement variation may be introduced by scan repetitions, repositioning at different time points, and image post-processing. Moreover, 3T may be susceptible to variation associated with increased field strength [8]. Such variability may pose limitations when conducting clinical comparisons to differentiate normal and diseased brains or in developing statistically predictive algorithms.To validate high resolution MT for detecting early disease or for monitoring progression in chronic neurological disease, it is necessary to collect information on normative values and to evaluate the reliability and reproducibility of the measurements when measured across time in healthy controls. This investigation evaluated observer-agreement of high-resolution MT measurements determined from repeated brain scans of 9 healthy volunteers. We postulated that MT values would remain stable during the one month study interval. We evaluated the reliability and reproducibility of the high resolution MT measurements in 12 brain regions of interest (ROIs), applied statistical measures to the data and used complex multivariate mixed-effects models to test the statistical significance of several effects due to region, subject, observer, time, and manual repetition.


## 2. Materials and Methods

### 2.1. Study Subjects

The study was approved by the IRB at the North Shore University Health System, and conducted following the ethical principles outlined in the Declaration of Helsinki. Eleven healthy adult volunteers were randomly selected from a database maintained at the Center for Advanced Imaging, Radiology Department, NorthShore University Health System provided written informed consent and evaluated for eligibility criteria. To protect the subjects’ confidentiality, all data were de-identified and handled according to the guidelines specified by the Health Insurance Portability and Accountability Act (HIPAA) in the USA.


### 2.2. Image Acquisition

Brain images were acquired using a 3T General Electric (GE) HDx system (Waukesha, WI, USA). Each volunteer was scanned twice in a randomly-selected time interval between 1 to 4 weeks. Methods for reducing random errors in image acquisition included the use of a body-coil for excitation to control B1 non-uniformities and an 8-channel quadrature receive-only coil [9]. MT pulses with (Ms) and without saturation (M0) were applied at an offset frequency from water resonance. To accelerate the scan for whole-brain coverage, while maintaining thin slices, the image protocol was optimized based on 3T using 3D SPGR [5]. The Gaussian Sinc MT pulse was applied in 8 ms at a 1200 HZ offset. The stability of the scanner and set-up procedure were addressed with a fixed set of parameters per subject. MT pulse was based on a three-dimensional spoiled gradient recalled (3D SPGR) acquisition. The image protocol included the following parameters: TR 34 to 35 ms, TE 4 to 8 ms, imaging FA 5°, bandwidth 15.6 kHz, 0.75 NEX, phase FOV 0.75, voxel dimensions 0.9 × 0.9 × 0.9 ~ 1.3 mm3. The whole brain was covered in 90 to 140 slices with acquisition time ranging from 7 minutes 40 seconds to 10 minutes 20 seconds using a partial k-space acquisition.


### 2.3. Image Analysis

MTR maps were generated off-line on a General Electric AW Workstation (General Electric, Milwaukee, WI, USA) using the standard equation:(1)MTR=M0-MSM0×100%,
where MS and M0 were the signal intensities in a given voxel obtained, with and without the MT saturation pulse, respectively. MTR maps generated based on the high resolution MT are demonstrated in Figure 1. The 12 ROIs were: genu, splenium, left and right hemispheres of the hippocampus, caudate, putamen, thalamus, and cerebral white matter. Figure 2 illustrated the 12 ROIs that were investigated. Each ROI was sized approximately 30 to 43 mm2 and manually and independently placed by Observers 1 and 2 (Authors S.S. and Y.W.) following procedures in classical and standard agreement studies [10]. After an initial consensus decision was drawn regarding the sizes and locations of the 12 ROIs, the observers performed manual segmentations of the ROI independently on each set of images. This ROI placement procedure was repeated by each observer in the following week.Figure 1
High resolution three-dimensional MTR map displayed both for the original view of the Axial plane and the reconstructed view of the Coronal plan. The MTR maps have excellent tissue conspicuity and high image resolution in all three dimensions.The Axial (a) and Coronal (b) views of high resolution MTR maps. Twelve brain ROIs are illustrated (white dots).
(a)(b)MTR values were extracted using the manually-defined ROIs with the combinations of observer, time point, and repetition (Table1). The mean and SDs of the ROI values were calculated. Meta-data were stored in a SAS 9.1 (SAS, Cary, NC, USA) dataset, with individual volunteer identification numbers withheld and replaced by a sequence of 1 to 9 for each subject.Table 1
The random or fixed effects in the data structure for the repeated measures MT study.
Outcome VariableYijklnEffect in the Variance-Component AnalysisType of EffectMathematical SymbolIndexMaximum of the IndexMean ROI Value via Manual SegmentationsSubjectRandomSii=i,…,II=9ObserverFixed or RandomOjj=1,…,JJ=2Time PointFixed or RandomTkk=1,…,KK=2RepetitionFixed or RandomRll=1,…,LL=2Region of InterestFixedKmm=1,…,MM=12Interaction TermsGenerally Mixed{Si;Oj;Tk;Rl;Km}{i;j;k;l;n}Based on the Appropriate Model SpecificationTable 2
Various strengths of correlation coefficients as a measure of concordance.
Absolute Value of the Correlation CoefficientStrength of the Concordance Between Samples0.0No0.2Weak0.5Moderate0.8Strong1.0Perfect


### 2.4. Statistical Methods

Statistical analyses were performed using SAS 9.1 (SAS Institute, Cary, NC, USA;http://www.sas.com). The SAS analytic procedures conducted included “Proc Univariate,” “Proc Means,” “Proc Corr,” and “Proc Mixed.” Bar diagrams were constructed using Microsoft Excel (http://www.microsoft.com). Age and gender were not controlled for in analyses.


#### 2.4.1. Descriptive Statistics

LetY=Yijklm having the indices described in Table 1 be a random variable representing the mean ROI value. For the mth ROI, we first computed the sample mean and standard deviation of all mean ROI values:(2)Mean̂(Ym)=Y••••m¯=1Nm∑l=12∑k=12∑j=12∑i=19Yijklm,SD̂(Ym)={Var̂(Ym)}1/2={1Nm-1∑l=12∑k=12∑j=12∑i=19(Yijklm-Y••••m¯)2}1/2,
where Nm = I×J×K×L = 9×23 = 72 measurements and the operator “•” means the marginal sum over the particular index.The 95-percentile normality range was approximately within the following interval, with the following lower and upper bounds:(3)(Mean̂(Ym)-2×SD̂(Ym),Mean̂(Ym)+2×SD̂(Ym)).
The term “normality range” as used in Europe, could be arbitrarily-defined according to the number of standard deviations away from the mean [11]. Thus, it should not be viewed as the range of the entire dataset, but rather an interval useful for estimating the population value by one or several standard deviations away from the mean. Here the critical value of 2 was chosen as recommended by Bland and Altman [12].Additionally, we justified using a Student’st-distribution with Nm-1 = 71 degrees of freedom. For any tail probability of α/2 (e.g., 0.025 for a 95-percent normality range), we used the quantile of the corresponding to particular t-distribution, such that(4)tNm-1-1(1-α2)=t71-1(0.975)=1.994,
This value happened to be close to the recommended multiplier of 2. Therefore, we rounded it to 2 in (3) for convenience.


#### 2.4.2. Concordance Using Spearman’s Rank Coefficient Coefficients

We first explored and measured the concordance between the various measurements fully nonparametrically via Spearman’s rank correlation coefficient. Suppose that we correlated the ROI values by Observersj = 1 and j′ = 2, then denoted the marginal ranks, Rijklm=ranki(Yijklm) andRij′klm=ranki′(Yi′jklm), respectively, for all j≠j′ with j = 1 and j′ = 2. The sample version of Pearson’s product-moment correlation coefficient between the ranks of the data was equivalent to Spearman’s rank correlation coefficient [13]:(5)Cor̂(rijklm,rij′klm)=(Nm/2)∑l=12∑k=12∑i=19(Ri1klmRi2klm)-∑l=12∑k=12∑i=19Ri1klm∑l=12∑k=12∑i=19Ri2klm{(Nm/2)∑l=12∑k=12∑i=19Ri1klm2-𝒜}1/2{(Nm/2)∑l=12∑k=12∑i=19Ri2klm2-ℬ}1/2,=∑l=12∑k=12∑i=19(Ri1klmRi2klm)-(Nm/2)Ri1klm¯Ri2klm¯(Nm/2-1)SD(Ri1klm)SD(Ri2klm).
where 𝒜 denotes (∑l=12∑k=12∑i=19Ri1klm)2 and ℬ denotes (∑l=12∑k=12∑i=19Ri2klm)2.Assuming that there was no presence of any ties since the ROI values were of continuous random variables, the Spearman’s rank correlation coefficient between Observersj and j′ was
(6)Corr(rijklm,rij′klm)=1-6∑l=12∑k=12∑i=19Di•klm2(Nm/2)(Nm2/4-1),
where the difference of an arbitrary pair of marginal ranks for Observer j and j′ was denoted by Di•klm=Rijklm-Rij′klm, for all j≠j′. Consequently, all of the raw mean ROI values were converted to their marginal ranks and the differences between the ranks of each observation on the two variables were computed. Spearman’s rank correlation coefficient was also computed for the ROI values between any two different time points k = 1 and k′ = 2.The strength of the concordance and the benchmark values have been discussed [14]. Bar diagrams were made to display the Spearman’s rank correlation coefficients between observers or time points for each ROI.


#### 2.4.3. Reproducibility Using Coefficients of Variations

We used the normalized measure of dispersion of a distribution to evaluate the reproducibility of the measurement [15]. The measure was the coefficient of variation (CV), defined as the ratio of the SD to the mean.(7)CV̂(Ym)=SD̂(Ym)Mean̂(Ym),
where both the numerator (i.e., sample SD) and the denominators (i.e., sample mean) in the above expression for CV are provided in (2). Skewed data, such as those generated by an exponential distribution for which the underlying population mean and standard deviation would be equal, and thus the CV became 1. Hence, CV < 1 would generally represent low variability, and CV > 1 would represent high variability. As in (4) and (6), further stratified computations of CV for different observers, time point, or repetitions were achieved using formulae similar to (7).


#### 2.4.4. Normality and Significance Tests for the Effects via a Multivariate Regression Analysis

As overall variability was likely a result of the effects illustrated in Table1. We employed a multivariate mixed-effects regression analysis to direct model the ROI values.A variance-component approach has advantages over many stratified analyses, especially studying studies with a limited sample size. Here, because of the novel imaging modality using MT and 3T acquisitions with labor-intensive manual segmentation procedures, large number of subjects would not have been feasible. To conduct an analysis of variance (ANOVA) based on the various effects, a distributional assumption of normality was necessary and convenient. Therefore, we conducted marginal normality tests using the Shapiro-Wilk test [16]. We would demonstrate (see Section 3.4) that the normality assumption was generally satisfactory.Thus, we could then consider adopting a linear random-effects model with all pair-wise interactions, in addition to a third-order interaction term:(8)Yijklm=μm+Si+Oj+Tk+Rl+Si×Oj+Si×Tk+Si×Rj+Oj×Tk+Oj×Rl+Tk×Rj+Oj×Tk×Rl+εijklm,∀i=1,…,9,j=1,2,k=1,2,l=1,2.
The effects represented the following: μm as intercept, Si as subjects, Oi as observers, Ti as time points, Ri as repetitions, and εijklm as the error team. A random-effects model assumed that each of the effects would have independent normal distributions with mean and variance.If normality had failed and because the data were mean ROI values that were positively-valued, we would recommend a Box-Cox transformation,h(Yijklm,λ), of the outcome variable with an optimal power coefficient λ [17–19]. Note that the log-normal becomes a special case when the power coefficient λ=0. This normality transformation is given by:
(9)Yijklm′=h(Yijklm,λ)={Yijklmλ-1λ,λ≠0log(Yijklmλ),λ=0∀i=1,…,9,j=1,2,k=1,2,l=1,2.A profile log-likelihood, llik ofλ given the observations yijklm, would be maximized to estimate an optimal Box-Cox transformation via a nonlinear minimization routine, where the log-likelihood was
(10)llik(λ∣yijklm)=-Nmlog{SD(yijklm′)}+(λ-1){∑i=1Nmlog(yijklm′)}+c,
where c was a constant free of the power coefficient to be optimized.Due to the limited number of subjects, however, even with an optimal normality transformation, over-fitting and non-convergence might be issues. Alternatively, we could regard all of the observers, time points, and repetitions as fixed and specify a mixed-effects model. The significances of the sources of variability were tested via a restricted maximum likelihood (REML) approach. For our multivariate analysis, the significance threshold for two-tailedP-values was set if P≤.05.


#### 2.4.5. Interobserver Reliability Using the ICCs

Stratified by the time points within each ROI, a two-way ANOVA was performed by regarding all of the observers, time points, and repetitions as fixed. We specified a mixed-effects model for simplicity. Due to the complexity of the variance components, we instead adopted a hybrid approach by considering two effects at once. For example, all subjects were segmented by the same observers who were from an entire population of observers. In other words, the subject effect was always assumed to be random, while the remaining effect (e.g., here the observer) was assumed to be fixed. We computed the Case-3 ICCs, accordingly [20].We simplified our notations by only keeping the indices for the subject and observer effects of interest. We decomposed the data as follows:(11)Yij=μ+Si+oj+Si×oj+εij,∀i=1,…,9,j=1,2,
where the subject effect Si was assumed to be random in an upper-case letter, which had a normal distribution with mean 0 and variance σS2, for all i=1,…,I (here I=9); the observer effect oj was considered to be a fixed effect in a lower-case letter, with the constraint ∑j=1Joj=0, with the corresponding parameter to the variance being θo2=(1/(J-1))∑j=1Joj2, for all j=1,…,J (here J=2); the interaction term between the subject and the observer Si×oj was the degree to which the jth observer departed from his or her usual rating tendencies for the ith subject, which had a normal distribution with a mean of 0 and variance σS×o2;  the errors terms εij were assumed to have an independent and identical distribution (iid) normal distribution with a mean of 0 and variance σE2. For the same ith subject, the effects are further assumed to be subjected to the constraint ∑j=1J(S×o)ij=0 over all of the observers. The corresponding two-way ANOVA table was listed (Table 3).Table 3
Two-way ANOVA table for the mixed-effects model.
Source of VariationDegrees of FreedomMean Squares(A) Between SubjectsI-1BSMSJσS2+σE2(B) Within SubjectsI(J-1)WSMSθo2+JσS×o2/(J-1)+σE2(B.1) Between ObserversJ-1OMSIθo2+JσS×o2/(J-1)+σE2(B.2) Error(I-1)(J-1)EMSJσS×o2/(J-1)+σE2Note: BSMS: Between Subjects Mean Squares; WSMS: Within Subject Mean Squares; OMS: Observer Mean Squares; EMS: Error Mean Squares.Shrout and Fleiss gave the true definition of ICC using the variance ratio of the subject variance over the total variance, with its estimated version using the quantities via ANOVA (Table3) [19]:(12)ICC=σS2-σS×o2/(J-1)σS2+σS×o2+σE2,ICĈ(3,1)=BSMS-EMSBSMS+(J-1)EMS.


#### 2.4.6. Intraobserver Reliability Using the ICCs

Similar to the analysis described above, we adopted a hybrid approach by considering two effects at once, with the subject effect always assumed to be random and the time point assumed to be fixed. The associate model was given by(13)Yij=μ+Si+tk+Si×tk+εik,∀i=1,…,9;k=1,2.As in (12), the estimated intraobserver agreement and its estimate were provided by:
(14)ICC=σS2-σS×t2/(K-1)σS2+σS×t2+σE2,ICĈ(3,1)=BSMS-EMSBSMS+(K-1)EMS,
where the interaction term the interaction term between the subject and the time Si×tk had a normal distribution with a mean of 0 and variance σS×t2.


#### 2.4.7. Sensitivity Analyses of the ICCs under Various Models

We performed a sensitivity analysis by computing 6 different ICC values Shrout and Fleiss previously proposed assumptions for ICCs (Table4) [18]. A SAS macro, written by Professor Robert Hamer, University of North Carolina School of Medicine, Chapel Hill, NC, USA (http://www.bios.unc.edu/~hamer), was run to perform the various ICC computations.Table 4
Six different ICCs computed via a sensitivity analysis of the modeling choices.
Notation for the ICC MeasureMultivariate Modeling AssumptionsICC(1,1)Each subject is rated by multiple observers; the observers are assumed to be randomly assigned to the subjects; all subjects have the same number of observers.ICC(2,1)All subjects are rated by the same observers who are assumed to be a random subset of all possible observers.ICC(3,1)All subjects are rated by the same observers who are assumed to be the entire population of observers.ICC(1,2)Same assumptions as ICC(1,1) but reliability for the mean of 2 ratings.ICC(2,2)Same assumptions as ICC(2,1) but reliability for the mean of 2 ratings.ICC(3,2)Same assumptions as ICC(3,1) but reliability for the mean of 2 ratings.  Assumes additionally there is no subject× observer interaction.


## 2.1. Study Subjects

The study was approved by the IRB at the North Shore University Health System, and conducted following the ethical principles outlined in the Declaration of Helsinki. Eleven healthy adult volunteers were randomly selected from a database maintained at the Center for Advanced Imaging, Radiology Department, NorthShore University Health System provided written informed consent and evaluated for eligibility criteria. To protect the subjects’ confidentiality, all data were de-identified and handled according to the guidelines specified by the Health Insurance Portability and Accountability Act (HIPAA) in the USA.


## 2.2. Image Acquisition

Brain images were acquired using a 3T General Electric (GE) HDx system (Waukesha, WI, USA). Each volunteer was scanned twice in a randomly-selected time interval between 1 to 4 weeks. Methods for reducing random errors in image acquisition included the use of a body-coil for excitation to control B1 non-uniformities and an 8-channel quadrature receive-only coil [9]. MT pulses with (Ms) and without saturation (M0) were applied at an offset frequency from water resonance. To accelerate the scan for whole-brain coverage, while maintaining thin slices, the image protocol was optimized based on 3T using 3D SPGR [5]. The Gaussian Sinc MT pulse was applied in 8 ms at a 1200 HZ offset. The stability of the scanner and set-up procedure were addressed with a fixed set of parameters per subject. MT pulse was based on a three-dimensional spoiled gradient recalled (3D SPGR) acquisition. The image protocol included the following parameters: TR 34 to 35 ms, TE 4 to 8 ms, imaging FA 5°, bandwidth 15.6 kHz, 0.75 NEX, phase FOV 0.75, voxel dimensions 0.9 × 0.9 × 0.9 ~ 1.3 mm3. The whole brain was covered in 90 to 140 slices with acquisition time ranging from 7 minutes 40 seconds to 10 minutes 20 seconds using a partial k-space acquisition.


## 2.3. Image Analysis

MTR maps were generated off-line on a General Electric AW Workstation (General Electric, Milwaukee, WI, USA) using the standard equation:(1)MTR=M0-MSM0×100%,
where MS and M0 were the signal intensities in a given voxel obtained, with and without the MT saturation pulse, respectively. MTR maps generated based on the high resolution MT are demonstrated in Figure 1. The 12 ROIs were: genu, splenium, left and right hemispheres of the hippocampus, caudate, putamen, thalamus, and cerebral white matter. Figure 2 illustrated the 12 ROIs that were investigated. Each ROI was sized approximately 30 to 43 mm2 and manually and independently placed by Observers 1 and 2 (Authors S.S. and Y.W.) following procedures in classical and standard agreement studies [10]. After an initial consensus decision was drawn regarding the sizes and locations of the 12 ROIs, the observers performed manual segmentations of the ROI independently on each set of images. This ROI placement procedure was repeated by each observer in the following week.Figure 1
High resolution three-dimensional MTR map displayed both for the original view of the Axial plane and the reconstructed view of the Coronal plan. The MTR maps have excellent tissue conspicuity and high image resolution in all three dimensions.The Axial (a) and Coronal (b) views of high resolution MTR maps. Twelve brain ROIs are illustrated (white dots).
(a)(b)MTR values were extracted using the manually-defined ROIs with the combinations of observer, time point, and repetition (Table1). The mean and SDs of the ROI values were calculated. Meta-data were stored in a SAS 9.1 (SAS, Cary, NC, USA) dataset, with individual volunteer identification numbers withheld and replaced by a sequence of 1 to 9 for each subject.Table 1
The random or fixed effects in the data structure for the repeated measures MT study.
Outcome VariableYijklnEffect in the Variance-Component AnalysisType of EffectMathematical SymbolIndexMaximum of the IndexMean ROI Value via Manual SegmentationsSubjectRandomSii=i,…,II=9ObserverFixed or RandomOjj=1,…,JJ=2Time PointFixed or RandomTkk=1,…,KK=2RepetitionFixed or RandomRll=1,…,LL=2Region of InterestFixedKmm=1,…,MM=12Interaction TermsGenerally Mixed{Si;Oj;Tk;Rl;Km}{i;j;k;l;n}Based on the Appropriate Model SpecificationTable 2
Various strengths of correlation coefficients as a measure of concordance.
Absolute Value of the Correlation CoefficientStrength of the Concordance Between Samples0.0No0.2Weak0.5Moderate0.8Strong1.0Perfect


## 2.4. Statistical Methods

Statistical analyses were performed using SAS 9.1 (SAS Institute, Cary, NC, USA;http://www.sas.com). The SAS analytic procedures conducted included “Proc Univariate,” “Proc Means,” “Proc Corr,” and “Proc Mixed.” Bar diagrams were constructed using Microsoft Excel (http://www.microsoft.com). Age and gender were not controlled for in analyses.


### 2.4.1. Descriptive Statistics

LetY=Yijklm having the indices described in Table 1 be a random variable representing the mean ROI value. For the mth ROI, we first computed the sample mean and standard deviation of all mean ROI values:(2)Mean̂(Ym)=Y••••m¯=1Nm∑l=12∑k=12∑j=12∑i=19Yijklm,SD̂(Ym)={Var̂(Ym)}1/2={1Nm-1∑l=12∑k=12∑j=12∑i=19(Yijklm-Y••••m¯)2}1/2,
where Nm = I×J×K×L = 9×23 = 72 measurements and the operator “•” means the marginal sum over the particular index.The 95-percentile normality range was approximately within the following interval, with the following lower and upper bounds:(3)(Mean̂(Ym)-2×SD̂(Ym),Mean̂(Ym)+2×SD̂(Ym)).
The term “normality range” as used in Europe, could be arbitrarily-defined according to the number of standard deviations away from the mean [11]. Thus, it should not be viewed as the range of the entire dataset, but rather an interval useful for estimating the population value by one or several standard deviations away from the mean. Here the critical value of 2 was chosen as recommended by Bland and Altman [12].Additionally, we justified using a Student’st-distribution with Nm-1 = 71 degrees of freedom. For any tail probability of α/2 (e.g., 0.025 for a 95-percent normality range), we used the quantile of the corresponding to particular t-distribution, such that(4)tNm-1-1(1-α2)=t71-1(0.975)=1.994,
This value happened to be close to the recommended multiplier of 2. Therefore, we rounded it to 2 in (3) for convenience.


### 2.4.2. Concordance Using Spearman’s Rank Coefficient Coefficients

We first explored and measured the concordance between the various measurements fully nonparametrically via Spearman’s rank correlation coefficient. Suppose that we correlated the ROI values by Observersj = 1 and j′ = 2, then denoted the marginal ranks, Rijklm=ranki(Yijklm) andRij′klm=ranki′(Yi′jklm), respectively, for all j≠j′ with j = 1 and j′ = 2. The sample version of Pearson’s product-moment correlation coefficient between the ranks of the data was equivalent to Spearman’s rank correlation coefficient [13]:(5)Cor̂(rijklm,rij′klm)=(Nm/2)∑l=12∑k=12∑i=19(Ri1klmRi2klm)-∑l=12∑k=12∑i=19Ri1klm∑l=12∑k=12∑i=19Ri2klm{(Nm/2)∑l=12∑k=12∑i=19Ri1klm2-𝒜}1/2{(Nm/2)∑l=12∑k=12∑i=19Ri2klm2-ℬ}1/2,=∑l=12∑k=12∑i=19(Ri1klmRi2klm)-(Nm/2)Ri1klm¯Ri2klm¯(Nm/2-1)SD(Ri1klm)SD(Ri2klm).
where 𝒜 denotes (∑l=12∑k=12∑i=19Ri1klm)2 and ℬ denotes (∑l=12∑k=12∑i=19Ri2klm)2.Assuming that there was no presence of any ties since the ROI values were of continuous random variables, the Spearman’s rank correlation coefficient between Observersj and j′ was
(6)Corr(rijklm,rij′klm)=1-6∑l=12∑k=12∑i=19Di•klm2(Nm/2)(Nm2/4-1),
where the difference of an arbitrary pair of marginal ranks for Observer j and j′ was denoted by Di•klm=Rijklm-Rij′klm, for all j≠j′. Consequently, all of the raw mean ROI values were converted to their marginal ranks and the differences between the ranks of each observation on the two variables were computed. Spearman’s rank correlation coefficient was also computed for the ROI values between any two different time points k = 1 and k′ = 2.The strength of the concordance and the benchmark values have been discussed [14]. Bar diagrams were made to display the Spearman’s rank correlation coefficients between observers or time points for each ROI.


### 2.4.3. Reproducibility Using Coefficients of Variations

We used the normalized measure of dispersion of a distribution to evaluate the reproducibility of the measurement [15]. The measure was the coefficient of variation (CV), defined as the ratio of the SD to the mean.(7)CV̂(Ym)=SD̂(Ym)Mean̂(Ym),
where both the numerator (i.e., sample SD) and the denominators (i.e., sample mean) in the above expression for CV are provided in (2). Skewed data, such as those generated by an exponential distribution for which the underlying population mean and standard deviation would be equal, and thus the CV became 1. Hence, CV < 1 would generally represent low variability, and CV > 1 would represent high variability. As in (4) and (6), further stratified computations of CV for different observers, time point, or repetitions were achieved using formulae similar to (7).


### 2.4.4. Normality and Significance Tests for the Effects via a Multivariate Regression Analysis

As overall variability was likely a result of the effects illustrated in Table1. We employed a multivariate mixed-effects regression analysis to direct model the ROI values.A variance-component approach has advantages over many stratified analyses, especially studying studies with a limited sample size. Here, because of the novel imaging modality using MT and 3T acquisitions with labor-intensive manual segmentation procedures, large number of subjects would not have been feasible. To conduct an analysis of variance (ANOVA) based on the various effects, a distributional assumption of normality was necessary and convenient. Therefore, we conducted marginal normality tests using the Shapiro-Wilk test [16]. We would demonstrate (see Section 3.4) that the normality assumption was generally satisfactory.Thus, we could then consider adopting a linear random-effects model with all pair-wise interactions, in addition to a third-order interaction term:(8)Yijklm=μm+Si+Oj+Tk+Rl+Si×Oj+Si×Tk+Si×Rj+Oj×Tk+Oj×Rl+Tk×Rj+Oj×Tk×Rl+εijklm,∀i=1,…,9,j=1,2,k=1,2,l=1,2.
The effects represented the following: μm as intercept, Si as subjects, Oi as observers, Ti as time points, Ri as repetitions, and εijklm as the error team. A random-effects model assumed that each of the effects would have independent normal distributions with mean and variance.If normality had failed and because the data were mean ROI values that were positively-valued, we would recommend a Box-Cox transformation,h(Yijklm,λ), of the outcome variable with an optimal power coefficient λ [17–19]. Note that the log-normal becomes a special case when the power coefficient λ=0. This normality transformation is given by:
(9)Yijklm′=h(Yijklm,λ)={Yijklmλ-1λ,λ≠0log(Yijklmλ),λ=0∀i=1,…,9,j=1,2,k=1,2,l=1,2.A profile log-likelihood, llik ofλ given the observations yijklm, would be maximized to estimate an optimal Box-Cox transformation via a nonlinear minimization routine, where the log-likelihood was
(10)llik(λ∣yijklm)=-Nmlog{SD(yijklm′)}+(λ-1){∑i=1Nmlog(yijklm′)}+c,
where c was a constant free of the power coefficient to be optimized.Due to the limited number of subjects, however, even with an optimal normality transformation, over-fitting and non-convergence might be issues. Alternatively, we could regard all of the observers, time points, and repetitions as fixed and specify a mixed-effects model. The significances of the sources of variability were tested via a restricted maximum likelihood (REML) approach. For our multivariate analysis, the significance threshold for two-tailedP-values was set if P≤.05.


### 2.4.5. Interobserver Reliability Using the ICCs

Stratified by the time points within each ROI, a two-way ANOVA was performed by regarding all of the observers, time points, and repetitions as fixed. We specified a mixed-effects model for simplicity. Due to the complexity of the variance components, we instead adopted a hybrid approach by considering two effects at once. For example, all subjects were segmented by the same observers who were from an entire population of observers. In other words, the subject effect was always assumed to be random, while the remaining effect (e.g., here the observer) was assumed to be fixed. We computed the Case-3 ICCs, accordingly [20].We simplified our notations by only keeping the indices for the subject and observer effects of interest. We decomposed the data as follows:(11)Yij=μ+Si+oj+Si×oj+εij,∀i=1,…,9,j=1,2,
where the subject effect Si was assumed to be random in an upper-case letter, which had a normal distribution with mean 0 and variance σS2, for all i=1,…,I (here I=9); the observer effect oj was considered to be a fixed effect in a lower-case letter, with the constraint ∑j=1Joj=0, with the corresponding parameter to the variance being θo2=(1/(J-1))∑j=1Joj2, for all j=1,…,J (here J=2); the interaction term between the subject and the observer Si×oj was the degree to which the jth observer departed from his or her usual rating tendencies for the ith subject, which had a normal distribution with a mean of 0 and variance σS×o2;  the errors terms εij were assumed to have an independent and identical distribution (iid) normal distribution with a mean of 0 and variance σE2. For the same ith subject, the effects are further assumed to be subjected to the constraint ∑j=1J(S×o)ij=0 over all of the observers. The corresponding two-way ANOVA table was listed (Table 3).Table 3
Two-way ANOVA table for the mixed-effects model.
Source of VariationDegrees of FreedomMean Squares(A) Between SubjectsI-1BSMSJσS2+σE2(B) Within SubjectsI(J-1)WSMSθo2+JσS×o2/(J-1)+σE2(B.1) Between ObserversJ-1OMSIθo2+JσS×o2/(J-1)+σE2(B.2) Error(I-1)(J-1)EMSJσS×o2/(J-1)+σE2Note: BSMS: Between Subjects Mean Squares; WSMS: Within Subject Mean Squares; OMS: Observer Mean Squares; EMS: Error Mean Squares.Shrout and Fleiss gave the true definition of ICC using the variance ratio of the subject variance over the total variance, with its estimated version using the quantities via ANOVA (Table3) [19]:(12)ICC=σS2-σS×o2/(J-1)σS2+σS×o2+σE2,ICĈ(3,1)=BSMS-EMSBSMS+(J-1)EMS.


### 2.4.6. Intraobserver Reliability Using the ICCs

Similar to the analysis described above, we adopted a hybrid approach by considering two effects at once, with the subject effect always assumed to be random and the time point assumed to be fixed. The associate model was given by(13)Yij=μ+Si+tk+Si×tk+εik,∀i=1,…,9;k=1,2.As in (12), the estimated intraobserver agreement and its estimate were provided by:
(14)ICC=σS2-σS×t2/(K-1)σS2+σS×t2+σE2,ICĈ(3,1)=BSMS-EMSBSMS+(K-1)EMS,
where the interaction term the interaction term between the subject and the time Si×tk had a normal distribution with a mean of 0 and variance σS×t2.


### 2.4.7. Sensitivity Analyses of the ICCs under Various Models

We performed a sensitivity analysis by computing 6 different ICC values Shrout and Fleiss previously proposed assumptions for ICCs (Table4) [18]. A SAS macro, written by Professor Robert Hamer, University of North Carolina School of Medicine, Chapel Hill, NC, USA (http://www.bios.unc.edu/~hamer), was run to perform the various ICC computations.Table 4
Six different ICCs computed via a sensitivity analysis of the modeling choices.
Notation for the ICC MeasureMultivariate Modeling AssumptionsICC(1,1)Each subject is rated by multiple observers; the observers are assumed to be randomly assigned to the subjects; all subjects have the same number of observers.ICC(2,1)All subjects are rated by the same observers who are assumed to be a random subset of all possible observers.ICC(3,1)All subjects are rated by the same observers who are assumed to be the entire population of observers.ICC(1,2)Same assumptions as ICC(1,1) but reliability for the mean of 2 ratings.ICC(2,2)Same assumptions as ICC(2,1) but reliability for the mean of 2 ratings.ICC(3,2)Same assumptions as ICC(3,1) but reliability for the mean of 2 ratings.  Assumes additionally there is no subject× observer interaction.


## 2.4.1. Descriptive Statistics

LetY=Yijklm having the indices described in Table 1 be a random variable representing the mean ROI value. For the mth ROI, we first computed the sample mean and standard deviation of all mean ROI values:(2)Mean̂(Ym)=Y••••m¯=1Nm∑l=12∑k=12∑j=12∑i=19Yijklm,SD̂(Ym)={Var̂(Ym)}1/2={1Nm-1∑l=12∑k=12∑j=12∑i=19(Yijklm-Y••••m¯)2}1/2,
where Nm = I×J×K×L = 9×23 = 72 measurements and the operator “•” means the marginal sum over the particular index.The 95-percentile normality range was approximately within the following interval, with the following lower and upper bounds:(3)(Mean̂(Ym)-2×SD̂(Ym),Mean̂(Ym)+2×SD̂(Ym)).
The term “normality range” as used in Europe, could be arbitrarily-defined according to the number of standard deviations away from the mean [11]. Thus, it should not be viewed as the range of the entire dataset, but rather an interval useful for estimating the population value by one or several standard deviations away from the mean. Here the critical value of 2 was chosen as recommended by Bland and Altman [12].Additionally, we justified using a Student’st-distribution with Nm-1 = 71 degrees of freedom. For any tail probability of α/2 (e.g., 0.025 for a 95-percent normality range), we used the quantile of the corresponding to particular t-distribution, such that(4)tNm-1-1(1-α2)=t71-1(0.975)=1.994,
This value happened to be close to the recommended multiplier of 2. Therefore, we rounded it to 2 in (3) for convenience.


## 2.4.2. Concordance Using Spearman’s Rank Coefficient Coefficients

We first explored and measured the concordance between the various measurements fully nonparametrically via Spearman’s rank correlation coefficient. Suppose that we correlated the ROI values by Observersj = 1 and j′ = 2, then denoted the marginal ranks, Rijklm=ranki(Yijklm) andRij′klm=ranki′(Yi′jklm), respectively, for all j≠j′ with j = 1 and j′ = 2. The sample version of Pearson’s product-moment correlation coefficient between the ranks of the data was equivalent to Spearman’s rank correlation coefficient [13]:(5)Cor̂(rijklm,rij′klm)=(Nm/2)∑l=12∑k=12∑i=19(Ri1klmRi2klm)-∑l=12∑k=12∑i=19Ri1klm∑l=12∑k=12∑i=19Ri2klm{(Nm/2)∑l=12∑k=12∑i=19Ri1klm2-𝒜}1/2{(Nm/2)∑l=12∑k=12∑i=19Ri2klm2-ℬ}1/2,=∑l=12∑k=12∑i=19(Ri1klmRi2klm)-(Nm/2)Ri1klm¯Ri2klm¯(Nm/2-1)SD(Ri1klm)SD(Ri2klm).
where 𝒜 denotes (∑l=12∑k=12∑i=19Ri1klm)2 and ℬ denotes (∑l=12∑k=12∑i=19Ri2klm)2.Assuming that there was no presence of any ties since the ROI values were of continuous random variables, the Spearman’s rank correlation coefficient between Observersj and j′ was
(6)Corr(rijklm,rij′klm)=1-6∑l=12∑k=12∑i=19Di•klm2(Nm/2)(Nm2/4-1),
where the difference of an arbitrary pair of marginal ranks for Observer j and j′ was denoted by Di•klm=Rijklm-Rij′klm, for all j≠j′. Consequently, all of the raw mean ROI values were converted to their marginal ranks and the differences between the ranks of each observation on the two variables were computed. Spearman’s rank correlation coefficient was also computed for the ROI values between any two different time points k = 1 and k′ = 2.The strength of the concordance and the benchmark values have been discussed [14]. Bar diagrams were made to display the Spearman’s rank correlation coefficients between observers or time points for each ROI.


## 2.4.3. Reproducibility Using Coefficients of Variations

We used the normalized measure of dispersion of a distribution to evaluate the reproducibility of the measurement [15]. The measure was the coefficient of variation (CV), defined as the ratio of the SD to the mean.(7)CV̂(Ym)=SD̂(Ym)Mean̂(Ym),
where both the numerator (i.e., sample SD) and the denominators (i.e., sample mean) in the above expression for CV are provided in (2). Skewed data, such as those generated by an exponential distribution for which the underlying population mean and standard deviation would be equal, and thus the CV became 1. Hence, CV < 1 would generally represent low variability, and CV > 1 would represent high variability. As in (4) and (6), further stratified computations of CV for different observers, time point, or repetitions were achieved using formulae similar to (7).


## 2.4.4. Normality and Significance Tests for the Effects via a Multivariate Regression Analysis

As overall variability was likely a result of the effects illustrated in Table1. We employed a multivariate mixed-effects regression analysis to direct model the ROI values.A variance-component approach has advantages over many stratified analyses, especially studying studies with a limited sample size. Here, because of the novel imaging modality using MT and 3T acquisitions with labor-intensive manual segmentation procedures, large number of subjects would not have been feasible. To conduct an analysis of variance (ANOVA) based on the various effects, a distributional assumption of normality was necessary and convenient. Therefore, we conducted marginal normality tests using the Shapiro-Wilk test [16]. We would demonstrate (see Section 3.4) that the normality assumption was generally satisfactory.Thus, we could then consider adopting a linear random-effects model with all pair-wise interactions, in addition to a third-order interaction term:(8)Yijklm=μm+Si+Oj+Tk+Rl+Si×Oj+Si×Tk+Si×Rj+Oj×Tk+Oj×Rl+Tk×Rj+Oj×Tk×Rl+εijklm,∀i=1,…,9,j=1,2,k=1,2,l=1,2.
The effects represented the following: μm as intercept, Si as subjects, Oi as observers, Ti as time points, Ri as repetitions, and εijklm as the error team. A random-effects model assumed that each of the effects would have independent normal distributions with mean and variance.If normality had failed and because the data were mean ROI values that were positively-valued, we would recommend a Box-Cox transformation,h(Yijklm,λ), of the outcome variable with an optimal power coefficient λ [17–19]. Note that the log-normal becomes a special case when the power coefficient λ=0. This normality transformation is given by:
(9)Yijklm′=h(Yijklm,λ)={Yijklmλ-1λ,λ≠0log(Yijklmλ),λ=0∀i=1,…,9,j=1,2,k=1,2,l=1,2.A profile log-likelihood, llik ofλ given the observations yijklm, would be maximized to estimate an optimal Box-Cox transformation via a nonlinear minimization routine, where the log-likelihood was
(10)llik(λ∣yijklm)=-Nmlog{SD(yijklm′)}+(λ-1){∑i=1Nmlog(yijklm′)}+c,
where c was a constant free of the power coefficient to be optimized.Due to the limited number of subjects, however, even with an optimal normality transformation, over-fitting and non-convergence might be issues. Alternatively, we could regard all of the observers, time points, and repetitions as fixed and specify a mixed-effects model. The significances of the sources of variability were tested via a restricted maximum likelihood (REML) approach. For our multivariate analysis, the significance threshold for two-tailedP-values was set if P≤.05.


## 2.4.5. Interobserver Reliability Using the ICCs

Stratified by the time points within each ROI, a two-way ANOVA was performed by regarding all of the observers, time points, and repetitions as fixed. We specified a mixed-effects model for simplicity. Due to the complexity of the variance components, we instead adopted a hybrid approach by considering two effects at once. For example, all subjects were segmented by the same observers who were from an entire population of observers. In other words, the subject effect was always assumed to be random, while the remaining effect (e.g., here the observer) was assumed to be fixed. We computed the Case-3 ICCs, accordingly [20].We simplified our notations by only keeping the indices for the subject and observer effects of interest. We decomposed the data as follows:(11)Yij=μ+Si+oj+Si×oj+εij,∀i=1,…,9,j=1,2,
where the subject effect Si was assumed to be random in an upper-case letter, which had a normal distribution with mean 0 and variance σS2, for all i=1,…,I (here I=9); the observer effect oj was considered to be a fixed effect in a lower-case letter, with the constraint ∑j=1Joj=0, with the corresponding parameter to the variance being θo2=(1/(J-1))∑j=1Joj2, for all j=1,…,J (here J=2); the interaction term between the subject and the observer Si×oj was the degree to which the jth observer departed from his or her usual rating tendencies for the ith subject, which had a normal distribution with a mean of 0 and variance σS×o2;  the errors terms εij were assumed to have an independent and identical distribution (iid) normal distribution with a mean of 0 and variance σE2. For the same ith subject, the effects are further assumed to be subjected to the constraint ∑j=1J(S×o)ij=0 over all of the observers. The corresponding two-way ANOVA table was listed (Table 3).Table 3
Two-way ANOVA table for the mixed-effects model.
Source of VariationDegrees of FreedomMean Squares(A) Between SubjectsI-1BSMSJσS2+σE2(B) Within SubjectsI(J-1)WSMSθo2+JσS×o2/(J-1)+σE2(B.1) Between ObserversJ-1OMSIθo2+JσS×o2/(J-1)+σE2(B.2) Error(I-1)(J-1)EMSJσS×o2/(J-1)+σE2Note: BSMS: Between Subjects Mean Squares; WSMS: Within Subject Mean Squares; OMS: Observer Mean Squares; EMS: Error Mean Squares.Shrout and Fleiss gave the true definition of ICC using the variance ratio of the subject variance over the total variance, with its estimated version using the quantities via ANOVA (Table3) [19]:(12)ICC=σS2-σS×o2/(J-1)σS2+σS×o2+σE2,ICĈ(3,1)=BSMS-EMSBSMS+(J-1)EMS.


## 2.4.6. Intraobserver Reliability Using the ICCs

Similar to the analysis described above, we adopted a hybrid approach by considering two effects at once, with the subject effect always assumed to be random and the time point assumed to be fixed. The associate model was given by(13)Yij=μ+Si+tk+Si×tk+εik,∀i=1,…,9;k=1,2.As in (12), the estimated intraobserver agreement and its estimate were provided by:
(14)ICC=σS2-σS×t2/(K-1)σS2+σS×t2+σE2,ICĈ(3,1)=BSMS-EMSBSMS+(K-1)EMS,
where the interaction term the interaction term between the subject and the time Si×tk had a normal distribution with a mean of 0 and variance σS×t2.


## 2.4.7. Sensitivity Analyses of the ICCs under Various Models

We performed a sensitivity analysis by computing 6 different ICC values Shrout and Fleiss previously proposed assumptions for ICCs (Table4) [18]. A SAS macro, written by Professor Robert Hamer, University of North Carolina School of Medicine, Chapel Hill, NC, USA (http://www.bios.unc.edu/~hamer), was run to perform the various ICC computations.Table 4
Six different ICCs computed via a sensitivity analysis of the modeling choices.
Notation for the ICC MeasureMultivariate Modeling AssumptionsICC(1,1)Each subject is rated by multiple observers; the observers are assumed to be randomly assigned to the subjects; all subjects have the same number of observers.ICC(2,1)All subjects are rated by the same observers who are assumed to be a random subset of all possible observers.ICC(3,1)All subjects are rated by the same observers who are assumed to be the entire population of observers.ICC(1,2)Same assumptions as ICC(1,1) but reliability for the mean of 2 ratings.ICC(2,2)Same assumptions as ICC(2,1) but reliability for the mean of 2 ratings.ICC(3,2)Same assumptions as ICC(3,1) but reliability for the mean of 2 ratings.  Assumes additionally there is no subject× observer interaction.


## 3. Results

### 3.1. Descriptive Statistics

Eleven healthy adults provided written informed consent to be evaluated and 9 underwent brain scans. Mean age of participants who received scans was37.9±14.2 years; 7 participants were men and 2 were women.The mean ROI values varied across different region (Table5). The left and right hemispheres tended to yield similar results when the average over these healthy subjects was considered.Table 5
Descriptive statistics and 95-percentile normality range of mean ROI values.
Region of InterestDescriptive Statistics (Mean± SD)95% Normality Range (Mean± 2 × SD)Genu77.0± 1.075.0–79.0Splenium72.8± 1.569.9–75.7Left Hippocampus51.5± 2.546.6–56.4Left Caudate59.5± 2.255.2–63.8Left Putamen62.0± 2.058.1–65.9Left Thalamus61.6± 2.357.1–66.1Left Cerebral White Matter73.2± 1.270.8–75.6Right Hippocampus52.0± 3.345.5–58.5Right Caudate61.3± 1.758.0–64.6Right Putamen62.8± 1.559.9–65.7Right Thalamus61.1± 2.556.2–66.0Right Cerebral White Matter73.0± 1.370.5–75.5Note: Results were pooled among all 72 observations within each region of interest. SD: standard deviation.


### 3.2. Concordance Using Spearman’s Rank Coefficient Coefficients

Spearman’s rank correlation coefficients showed that a majority of correlations within each observer was above 0.5, suggesting a moderate to high concordance (Figure3). Time point 2 tended to yield higher concordance between the observers, which suggested a possible learning effect over time (Figure 4). Due to limited sample sizes in this pilot study, in Figures 3 and 4, we demonstrated the effect of observers by averaging over repetitions by each observer. Similarly, we demonstrated the effect of time points by averaging over repetitions at each time point.Figure 3
Spearman’s rank correlation coefficients between the two different time points for the same observer (red = Observer 1; blue = Observer 2).Figure 4
Spearman’s rank correlation coefficients between the two different observers for the same time point (orange = Time Point 1; green = Time Point 2).


### 3.3. Reproducibility Using Coefficients of Variations

Overall, CVs ranged from 1.2% in the genu for Observer 2 to 7.0% in the right hippocampus for Observer 1 (Table6). Since all of the CVs were within 7%, that is, all CVs were less than 10%, the reproducibility was reasonably high.Table 6
Coefficient of Variation (CV)  of the mean Region of Interest values for each observer.
Region of InterestObserver 1Observer 2Mean± SD (N=36)CV (%)Mean± SD (N=36)CV   (%)Genu76.9± 1.01.377.1± 0.91.2Splenium73.1± 1.41.972.6± 1.52.1Left Hippocampus51.3± 2.44.751.6± 2.75.2Left Caudate59.7± 1.93.259.3± 2.54.2Left Putamen61.9± 2.23.662.1± 1.93.1Left Thalamus59.9± 1.52.563.3± 1.72.7Left Cerebral White Matter73.3± 1.31.873.1± 1.21.6Right Hippocampus52.5± 3.77.051.5± 2.75.2Right Caudate61.2± 1.93.161.5± 1.42.3Right Putamen62.7± 1.52.462.8± 1.52.4Right Thalamus59.7± 1.72.862.5± 2.54.0Right Cerebral White Matter73.2± 1.21.672.8± 1.41.9Note. SD: standard deviation.


### 3.4. Normality and Significance Tests via a Multivariate Analysis

The tests of the normal distribution assumption marginally using the Shapiro-Wilk test indicated that only occasionally (e.g., for left caudate, left and right putamen, and right hippocampus), this assumption was not met (see Table7). Therefore, it was reasonable to specify linear mixed-effects modeling and two-way ANOVA reported in Sections 3.5 and 3.6.Table 7
P-value from the Shapiro-Wilk test of marginal normal distributions.
Region of InterestP-valueP-valueTime Point 1Time Point 2Observer 1Observer 2Observer 1Observer 2Genu.29.17.70.36Splenium.31.06.93.61Left Hippocampus.14.81.45>.99Left Caudate.97<.0001a.49.92Left Putamen.20.06.01a.01aLeft Thalamus.86.51.63.13Left Cerebral White Matter.82.43.21.02Right Hippocampus.54.86.01a.58Right Caudate.49.80.60.89Right Putamen.07.003a.25.03aRight Thalamus.50.68.82.13Right Cerebral White Matter.79.78.16.54aNormal distribution was not met.


### 3.5. Interobserver Reliability Using the ICCs

At time point 1, ICCs were greater than 0.7 in regions of genu, left and right putamen, whereas ICCs were from 0.5 to 0.7 in regions of splenium, left and right hippocampus, left caudate, and right cerebral white matter (Table8). These results indicated moderate to strong interobserver reliability. In comparison, at time point 2, ICCs were greater than 0.7 in regions of genu, splenium, left and right caudate, putamen and cerebral white matter, and left hippocampus and thalamus, while ICCs were from 0.5 to 0.7 in right hippocampus and thalamus. These results suggested a learning effect over time. However, for some ROIs such as the left cerebral white matter, right caudate, right thalamus, ICCs increased from 0.2 (at time point 1) to 0.9 (at time point 2), making it difficult to determine whether this represents a learning effect.Table 8
Interobserver reliability between two observers for each time point.
Region of InterestInter-Reader ICCInter-Reader ICCTime Point 1Time Point 2Genu0.8660.726Splenium0.5370.758Left Hippocampus0.6930.796Left Caudate0.5800.902Left Putamen0.8690.962Left Thalamus0.4100.855Left Cerebral White Matter0.3780.929Right Hippocampus0.6530.656Right Caudate0.2090.872Right Putamen0.7250.882Right Thalamus0.2640.572Right Cerebral White Matter0.6370.896


### 3.6. Intraobserver Reliability Using the ICCs

At each time point, intraobserver agreement was at least 0.5 for a majority of the regions (Table9).Table 9
Intraobserver reliability within each observer between different repetitions.
Region of InterestIntraobserver ICCIntraobserver ICCObserver 1Observer 2Genu0.5370.555Splenium0.5980.756Left Hippocampus0.5200.596Left Caudate0.7090.362Left Putamen0.9400.784Left Thalamus0.4790.622Left Cerebral White Matter0.5600.703Right Hippocampus0.4110.826Right Caudate0.4730.436Right Putamen0.6590.657Right Thalamus0.6870.308Right Cerebral White Matter0.5700.770


### 3.7. Sensitivity Analyses of the ICCs under Various Models

Six different methods for generating ICCs exhibited similar patterns for high vs. low reliability results in different ROIs (Table10). Thus, reliability appeared to be sensitive to ROI.Table 10
Sensitivity analysis of 6 different interobserver ICCs.
Region of InterestICC (1,1)ICC (2,1)ICC (3, 1)ICC (1, 2)ICC (2, 2)ICC (3, 2)Interobserver ICC at Time 1Genu0.8700.8790.8660.9310.9350.928Splenium0.4970.4630.5370.6640.6330.699Left Hippocampus0.6530.6050.6930.7900.7540.819Left Caudate0.5620.5420.5800.7190.7030.734Left Putamen0.8710.8740.8690.9310.9330.930Left Thalamus-0.0150.1140.410-0.0300.2050.581Left Cerebral White Matter0.3820.3850.3780.5530.5560.549Right Hippocampus0.6600.6690.6530.7950.8020.790Right Caudate0.1780.1800.2090.3020.3060.346Right Putamen0.7250.7320.7200.8400.8450.837Right Thalamus-0.0920.0790.264-0.2020.1460.417Right Cerebral White Matter0.6300.6210.6370.7730.7660.779Interobserver ICC at Time 2Genu0.7220.7150.7260.8380.8340.841Splenium0.7580.7570.7580.8620.8620.863Left Hippocampus0.7920.7850.7960.8840.8800.886Left Caudate0.9050.9090.9020.9500.9520.949Left Putamen0.9610.9590.9620.9800.9790.980Left Thalamus0.2970.2390.8550.4580.3850.922Left Cerebral White Matter0.9280.9260.9290.9630.9620.963Right Hippocampus0.6400.6200.6560.7810.7650.793Right Caudate0.8760.8840.8720.9340.9380.932Right Putamen0.8840.8870.8820.9380.9400.937Right Thalamus0.4190.3470.5720.5910.5160.728Right Cerebral White Matter0.8890.8760.8960.9410.9340.945Table 11
Sensitivity analysis of 6 different intraobserver ICCs.
Region of InterestICC (1,1)ICC (2,1)ICC (3, 1)ICC (1,k)ICC (2,k)ICC (3,k)Intraobserver for Observer 1Genu0.5370.5370.5370.6990.6990.699Splenium0.5900.5790.5980.7420.7330.749Left Hippocampus0.5310.5440.5200.6940.7050.684Left Caudate0.7040.6960.7090.8260.8210.830Left Putamen0.9420.9460.9400.9700.9720.969Left Thalamus0.4810.4840.4790.6500.6530.647Left Cerebral White Matter0.5500.5390.5600.7100.7010.718Right Hippocampus0.4260.4390.4110.5970.6100.582Right Caudate0.4700.4670.4730.6400.6370.643Right Putamen0.6570.6540.6590.7930.7910.795Right Thalamus0.6960.7110.6870.8210.8310.814Right Cerebral White Matter0.5820.5960.5700.7360.7470.727Intraobserver ICC for Observer 2Genu0.5630.5720.5550.7200.7280.714Splenium0.7600.7670.7560.8640.8680.861Left Hippocampus0.6070.6230.5960.7560.7670.747Left Caudate0.3650.3670.3620.5350.5370.531Left Putamen0.7900.8000.7840.8830.8890.879Left Thalamus0.6320.6450.6220.7740.7840.767Left Cerebral White Matter0.7120.7260.7030.8320.8410.826Right Hippocampus0.8290.8350.8260.9070.9100.905Right Caudate0.4320.4290.4360.6030.6010.607Right Putamen0.6670.6820.6570.8000.8110.793Right Thalamus0.2980.2940.3080.4590.4550.471Right Cerebral White Matter0.7770.7890.7700.8750.8820.870


## 3.1. Descriptive Statistics

Eleven healthy adults provided written informed consent to be evaluated and 9 underwent brain scans. Mean age of participants who received scans was37.9±14.2 years; 7 participants were men and 2 were women.The mean ROI values varied across different region (Table5). The left and right hemispheres tended to yield similar results when the average over these healthy subjects was considered.Table 5
Descriptive statistics and 95-percentile normality range of mean ROI values.
Region of InterestDescriptive Statistics (Mean± SD)95% Normality Range (Mean± 2 × SD)Genu77.0± 1.075.0–79.0Splenium72.8± 1.569.9–75.7Left Hippocampus51.5± 2.546.6–56.4Left Caudate59.5± 2.255.2–63.8Left Putamen62.0± 2.058.1–65.9Left Thalamus61.6± 2.357.1–66.1Left Cerebral White Matter73.2± 1.270.8–75.6Right Hippocampus52.0± 3.345.5–58.5Right Caudate61.3± 1.758.0–64.6Right Putamen62.8± 1.559.9–65.7Right Thalamus61.1± 2.556.2–66.0Right Cerebral White Matter73.0± 1.370.5–75.5Note: Results were pooled among all 72 observations within each region of interest. SD: standard deviation.


## 3.2. Concordance Using Spearman’s Rank Coefficient Coefficients

Spearman’s rank correlation coefficients showed that a majority of correlations within each observer was above 0.5, suggesting a moderate to high concordance (Figure3). Time point 2 tended to yield higher concordance between the observers, which suggested a possible learning effect over time (Figure 4). Due to limited sample sizes in this pilot study, in Figures 3 and 4, we demonstrated the effect of observers by averaging over repetitions by each observer. Similarly, we demonstrated the effect of time points by averaging over repetitions at each time point.Figure 3
Spearman’s rank correlation coefficients between the two different time points for the same observer (red = Observer 1; blue = Observer 2).Figure 4
Spearman’s rank correlation coefficients between the two different observers for the same time point (orange = Time Point 1; green = Time Point 2).


## 3.3. Reproducibility Using Coefficients of Variations

Overall, CVs ranged from 1.2% in the genu for Observer 2 to 7.0% in the right hippocampus for Observer 1 (Table6). Since all of the CVs were within 7%, that is, all CVs were less than 10%, the reproducibility was reasonably high.Table 6
Coefficient of Variation (CV)  of the mean Region of Interest values for each observer.
Region of InterestObserver 1Observer 2Mean± SD (N=36)CV (%)Mean± SD (N=36)CV   (%)Genu76.9± 1.01.377.1± 0.91.2Splenium73.1± 1.41.972.6± 1.52.1Left Hippocampus51.3± 2.44.751.6± 2.75.2Left Caudate59.7± 1.93.259.3± 2.54.2Left Putamen61.9± 2.23.662.1± 1.93.1Left Thalamus59.9± 1.52.563.3± 1.72.7Left Cerebral White Matter73.3± 1.31.873.1± 1.21.6Right Hippocampus52.5± 3.77.051.5± 2.75.2Right Caudate61.2± 1.93.161.5± 1.42.3Right Putamen62.7± 1.52.462.8± 1.52.4Right Thalamus59.7± 1.72.862.5± 2.54.0Right Cerebral White Matter73.2± 1.21.672.8± 1.41.9Note. SD: standard deviation.


## 3.4. Normality and Significance Tests via a Multivariate Analysis

The tests of the normal distribution assumption marginally using the Shapiro-Wilk test indicated that only occasionally (e.g., for left caudate, left and right putamen, and right hippocampus), this assumption was not met (see Table7). Therefore, it was reasonable to specify linear mixed-effects modeling and two-way ANOVA reported in Sections 3.5 and 3.6.Table 7
P-value from the Shapiro-Wilk test of marginal normal distributions.
Region of InterestP-valueP-valueTime Point 1Time Point 2Observer 1Observer 2Observer 1Observer 2Genu.29.17.70.36Splenium.31.06.93.61Left Hippocampus.14.81.45>.99Left Caudate.97<.0001a.49.92Left Putamen.20.06.01a.01aLeft Thalamus.86.51.63.13Left Cerebral White Matter.82.43.21.02Right Hippocampus.54.86.01a.58Right Caudate.49.80.60.89Right Putamen.07.003a.25.03aRight Thalamus.50.68.82.13Right Cerebral White Matter.79.78.16.54aNormal distribution was not met.


## 3.5. Interobserver Reliability Using the ICCs

At time point 1, ICCs were greater than 0.7 in regions of genu, left and right putamen, whereas ICCs were from 0.5 to 0.7 in regions of splenium, left and right hippocampus, left caudate, and right cerebral white matter (Table8). These results indicated moderate to strong interobserver reliability. In comparison, at time point 2, ICCs were greater than 0.7 in regions of genu, splenium, left and right caudate, putamen and cerebral white matter, and left hippocampus and thalamus, while ICCs were from 0.5 to 0.7 in right hippocampus and thalamus. These results suggested a learning effect over time. However, for some ROIs such as the left cerebral white matter, right caudate, right thalamus, ICCs increased from 0.2 (at time point 1) to 0.9 (at time point 2), making it difficult to determine whether this represents a learning effect.Table 8
Interobserver reliability between two observers for each time point.
Region of InterestInter-Reader ICCInter-Reader ICCTime Point 1Time Point 2Genu0.8660.726Splenium0.5370.758Left Hippocampus0.6930.796Left Caudate0.5800.902Left Putamen0.8690.962Left Thalamus0.4100.855Left Cerebral White Matter0.3780.929Right Hippocampus0.6530.656Right Caudate0.2090.872Right Putamen0.7250.882Right Thalamus0.2640.572Right Cerebral White Matter0.6370.896


## 3.6. Intraobserver Reliability Using the ICCs

At each time point, intraobserver agreement was at least 0.5 for a majority of the regions (Table9).Table 9
Intraobserver reliability within each observer between different repetitions.
Region of InterestIntraobserver ICCIntraobserver ICCObserver 1Observer 2Genu0.5370.555Splenium0.5980.756Left Hippocampus0.5200.596Left Caudate0.7090.362Left Putamen0.9400.784Left Thalamus0.4790.622Left Cerebral White Matter0.5600.703Right Hippocampus0.4110.826Right Caudate0.4730.436Right Putamen0.6590.657Right Thalamus0.6870.308Right Cerebral White Matter0.5700.770


## 3.7. Sensitivity Analyses of the ICCs under Various Models

Six different methods for generating ICCs exhibited similar patterns for high vs. low reliability results in different ROIs (Table10). Thus, reliability appeared to be sensitive to ROI.Table 10
Sensitivity analysis of 6 different interobserver ICCs.
Region of InterestICC (1,1)ICC (2,1)ICC (3, 1)ICC (1, 2)ICC (2, 2)ICC (3, 2)Interobserver ICC at Time 1Genu0.8700.8790.8660.9310.9350.928Splenium0.4970.4630.5370.6640.6330.699Left Hippocampus0.6530.6050.6930.7900.7540.819Left Caudate0.5620.5420.5800.7190.7030.734Left Putamen0.8710.8740.8690.9310.9330.930Left Thalamus-0.0150.1140.410-0.0300.2050.581Left Cerebral White Matter0.3820.3850.3780.5530.5560.549Right Hippocampus0.6600.6690.6530.7950.8020.790Right Caudate0.1780.1800.2090.3020.3060.346Right Putamen0.7250.7320.7200.8400.8450.837Right Thalamus-0.0920.0790.264-0.2020.1460.417Right Cerebral White Matter0.6300.6210.6370.7730.7660.779Interobserver ICC at Time 2Genu0.7220.7150.7260.8380.8340.841Splenium0.7580.7570.7580.8620.8620.863Left Hippocampus0.7920.7850.7960.8840.8800.886Left Caudate0.9050.9090.9020.9500.9520.949Left Putamen0.9610.9590.9620.9800.9790.980Left Thalamus0.2970.2390.8550.4580.3850.922Left Cerebral White Matter0.9280.9260.9290.9630.9620.963Right Hippocampus0.6400.6200.6560.7810.7650.793Right Caudate0.8760.8840.8720.9340.9380.932Right Putamen0.8840.8870.8820.9380.9400.937Right Thalamus0.4190.3470.5720.5910.5160.728Right Cerebral White Matter0.8890.8760.8960.9410.9340.945Table 11
Sensitivity analysis of 6 different intraobserver ICCs.
Region of InterestICC (1,1)ICC (2,1)ICC (3, 1)ICC (1,k)ICC (2,k)ICC (3,k)Intraobserver for Observer 1Genu0.5370.5370.5370.6990.6990.699Splenium0.5900.5790.5980.7420.7330.749Left Hippocampus0.5310.5440.5200.6940.7050.684Left Caudate0.7040.6960.7090.8260.8210.830Left Putamen0.9420.9460.9400.9700.9720.969Left Thalamus0.4810.4840.4790.6500.6530.647Left Cerebral White Matter0.5500.5390.5600.7100.7010.718Right Hippocampus0.4260.4390.4110.5970.6100.582Right Caudate0.4700.4670.4730.6400.6370.643Right Putamen0.6570.6540.6590.7930.7910.795Right Thalamus0.6960.7110.6870.8210.8310.814Right Cerebral White Matter0.5820.5960.5700.7360.7470.727Intraobserver ICC for Observer 2Genu0.5630.5720.5550.7200.7280.714Splenium0.7600.7670.7560.8640.8680.861Left Hippocampus0.6070.6230.5960.7560.7670.747Left Caudate0.3650.3670.3620.5350.5370.531Left Putamen0.7900.8000.7840.8830.8890.879Left Thalamus0.6320.6450.6220.7740.7840.767Left Cerebral White Matter0.7120.7260.7030.8320.8410.826Right Hippocampus0.8290.8350.8260.9070.9100.905Right Caudate0.4320.4290.4360.6030.6010.607Right Putamen0.6670.6820.6570.8000.8110.793Right Thalamus0.2980.2940.3080.4590.4550.471Right Cerebral White Matter0.7770.7890.7700.8750.8820.870


## 4. Conclusions and Discussion

We present mathematical methods for MT brain images using 3-T high resolution. Our image analysis may provide useful pilot information for future investigations. These mathematical and statistical methods may easily be generalized to practical studies with larger sample sizes or to studies of patients with active disease.We acquired repeat brain measurements based on a high resolution MT imaging protocol at 3T in 9 healthy adults. Our results indicate moderate to high reproducibility, supporting the validity of this method for further studies.  Overall, higher intraobserver reliability was observed at the second time point than that at the initial time point, suggesting a possible learning curve effect for both observers. Interobserver reliability was generally lower than intraobserver variability, suggesting a strong observer effect in this comparison, which may be a factor in future investigations using MT imaging.Our analyses examined different aspects in a typical observer-agreement study, using measures for concordance, reproducibility, reliability, variance-component analysis, and multivariate analysis. In other studies, all or some of such methods may be considered. However, with a simpler study of either several observers, or one observer with several repetitions at different sessions or time points, then these scenarios may only require several of our methods. Only a small sample of healthy volunteers was evaluated in this initial pilot study.  Therefore, the generalization of the 95-percentile normality range may be limited with respect to the wider spectrum of brain mechanisms represented in the broader population. For instance, demonstrating summary measures using all possible observer and time point combinations may not lead to meaningful interpretations in all cases. Nevertheless, since the technology is new, this research may provide useful pilot information for future investigations. Moreover, the statistical methods employed and illustrated here may easily be generalized to studies with larger sample sizes and diseased subjects.Another limitation was that this study aimed to evaluate only the reproducibility and reliability, rather than the accuracy in a more comprehensive validation study. In the absence of a true gold standard, such as one based on digital phantoms where realistic variability may still not be simulated, or on histopathology, improved reliability may not be equated with improved accuracy [21]. Both sensitivity and specificity are of interest. Further research would benefit from a useful algorithm to perhaps statistically and optimally estimate the underlying spatial “ground truth” [22, 23].Finally, future research may be directed to evaluating the diagnostic utility of high resolution MT for early detection of Alzheimer’s disease, multiple sclerosis or other neurological disorders and for monitoring progression across the clinical course.



---

*Source: 618747-2010-02-09.xml*