The effect of reducing terminal feedback frequency on the learning and performance of motor skills.
Creators: name: McKay, Brad
Publisher: name: Open Science Framework
Year: 2022
Subjects: subject: Sports Studies
subject: Social and Behavioral Sciences
subject: Psychology
subject: FOS: Psychology

description: 1. Working Title The effect of reducing terminal feedback frequency on the learning and performance of motor skills. 2. Anticipated Start Date &amp; Completion Date Start: August, 2020 End: January, 2021 3. Stage of Review at Completion of Pre-Registration Preliminary reading (COMPLETE) Preliminary scoping &amp; pilot search for studies (COMPLETE) Study Inclusion/Exclusion Criteria and Design (COMPLETE) Systematic Search &amp; Screening (in progress) Data Extraction (not yet started) Quality Assessment of Data (not yet started) Analysis of Data (not yet started) Project Write-Up (not yet started) 4. Research Objectives The motivation for this meta-analysis was to follow up on a seminal review conducted by Kantak and Winstein (2012) exploring the so-called learning-performance distinction in motor learning. In their review, Kantak and Winstein investigated the effect of feedback and practice schedule manipulations on immediate and delayed retention tests and concluded that effects often differed between tests. The present meta-analysis will endeavour to build on this research by accomplishing the objectives described below. This analysis will focus on the effect of a reduced terminal feedback frequency in order to plausibly estimate the true average effect of a specific type of intervention, in contrast to the original study which examined diverse intervention types in a qualitative manner. Feedback frequency was chosen for two reasons: Statistical power and theoretical basis. Studies examining reduced feedback schedules were the most prevalent sub-type of experiment analyzed in the K &amp; W review. Further, the guidance hypothesis (Salmoni, Schmidt, &amp; Walter, 1984) provides a theoretical basis for expecting the effect of a reduced feedback schedule to differ as a function of when retention is measured, and thus for the hypothetical ‘learning-performance distinction” proposed by K &amp; W. The present meta-analysis will investigate the effect of reducing terminal feedback frequency at immediate retention and delayed retention, and will investigate whether this effect significantly changes as a function of test time. Thus, the learning-performance distinction hypothesis will be investigated meta-analytically within the reduced feedback frequency literature. Further, this meta-analysis will re-examine the experiments included in the original K &amp; W review and explore the extent that publication bias may have influenced the results of that study. Primary Objectives: 1) Estimate the effect of providing a reduced frequency of terminal feedback on delayed retention of motor skills in a healthy population. 2) Estimate the same effect on the immediate retention of motor skills. 3) Estimate the same effect for the acquisition period. 4) Investigate whether the effect of reducing terminal feedback frequency changes from immediate to delayed retention. 5) Investigate whether the effect changes from acquisition to delayed retention. 6) Investigate the influence of publication bias on the primary meta-analysis. 7) Investigate the influence of publication bias on the results of Kantak &amp; Winstein (2012). Hypotheses to be tested: 1) Based on the guidance hypothesis (Salmoni, Schmidt, &amp; Walter, 1984): A reduced frequency of feedback during acquisition will result in superior performance on a delayed 24-hour retention test. 2) Based on the guidance hypothesis: A reduced frequency of feedback during acquisition will result in superior performance on an immediate, no feedback retention test. 3) Based on the guidance hypothesis: A 100% frequency of feedback will result in superior performance during acquisition. 4) 4) Based on the guidance hypothesis: The effect of feedback frequency will change from acquisition to delayed retention, such that 100% feedback is more effective for acquisition performance but less effective for delayed retention performance 5) Based on the Kantak &amp; Winstein motor memory paradigm: The effect of feedback frequency will change from immediate retention to delayed retention, such that the benefit of reduced feedback frequency will increase from immediate to delayed retention. 6) Based on our assessment of the motor learning literature: There will be evidence of significant selection effects around p = .025 (one-tailed), such that studies reporting statistically significant results will be overrepresented in the sample. 7) Based on our assessment of the motor learning literature: There will be evidence of significant selection effects around p = .025 (one-tailed) among the studies included in the Kantak &amp; Winstein review, such that studies reporting statistically significant results at delayed retention will be overrepresented in the sample. Secondary Objectives: Test the following potential moderators of the effect of reduced terminal feedback frequency on motor learning: Age group (Children, Adult, Older adult), Skill level (novice, experienced, expert), Task classification (based on Gentile’s 2 X 2 framework), Number of acquisition trials, Number of acquisition days, Frequency of terminal feedback, Publication status, Bandwidth provisioning, Faded feedback schedule. Hypotheses to be tested: 1) Based on the Challenge-Point Framework (Guadagnoli &amp; Lee, 2001): Children and older adults will perform more effectively on delayed retention tests after having practiced with 100% feedback during acquisition, while younger adults will perform more effectively after having received a reduced frequency of feedback. 2) Based on the Challenge-Point Framework: Higher skill performers will benefit from less feedback in acquisition than lower skilled learners when performance is measured on a delayed retention test. 3) Based on our assessment of the motor learning literature: Published articles will report significantly larger benefits of reduced feedback frequency when assessed at delayed retention. 4) Based on research comparing bandwidth feedback protocols to yoked groups: Providing feedback according to a bandwidth will have a larger benefit for delayed retention performance than reduced feedback frequency. 5) Based on the guidance hypothesis: A faded schedule of feedback during acquisition will be more effective than a static reduced schedule of feedback for delayed retention performance. Tertiary Objective: Examine the sensitivity of the primary analyses to model specification and other analysis decisions. 5. Inclusion criteria Experiments will be included based on the following inclusion criteria: 1. Published article or masters/doctorate thesis. 2. Experiment that employed random assignment to experimental conditions. 3. Included healthy, non-clinical participants. 4. Included immediate OR delayed retention test. 5. Included a feedback frequency manipulation and included a reference group receiving feedback after 100% of acquisition trials. 6. Written in English. 7. Studied a motor skill. 8. Included an objective measure of performance 6. Search Strategy Data-bases: PubMed, PsycInfo. Search query: “feedback” AND “motor learning” PubMed and PsycInfo databases were queried on August 4, 2020. Results from both searches were imported into Covidence for further screening. Covidence automatically identifies and removes duplicate entries and tracks these data on a PRIMSA-style flow chart. A total of 1990 studies were identified, of which 433 were duplicates. The resulting 1557 studies were screened based on title and abstract. Two researchers screened each article independently and decided if the article was potentially relevant to the present meta-analysis or not. Of the 1557 studies, there were 62 conflicts in which the researchers reached different decisions. Each conflict was resolved by a third researcher who was not involved in the first screening decision. Following title and abstract screening, 75 papers were identified for full text review. Again, each article will be reviewed by two researchers independently and conflicts in inclusion/exclusion decisions will be resolved by a third researcher. Additional articles will be sought by performing forward and backward reference searching. The reference sections of each included article will be searched for possible articles of interest. Further, each article will be found on Google Scholar and the “cited by” link will be searched for other potentially relevant articles. The final search strategy employed will be a targeted author search on Google Scholar. Any author with two or more included articles will be searched. 7. Potential Moderators a) Age group: Adult, children (under 18-years old), older adult (over 49-years-old) b) Skill level: Novice, experienced, expert (based on description in article). c) Task classification (based on Gentile’s 2 X 2 framework): Regulatory conditions (in motion, stable), Intertrial variability (with, without). d) Number of acquisition trials e) Number of acquisition days f) Frequency of terminal feedback in acquisition as a percentage of total trials. g) Publication status: published, unpublished. h) Bandwidth provision (yes, no) i) Faded feedback schedule (yes, no) 8. Primary Outcomes Measured (DVs) – Required. Objectively measured motor performance outcome on a delayed 24-hour retention test or immediate retention test. Only one outcome per time point per experiment will be included in the meta-analysis. The prioritization of motor performance outcomes for inclusion is based on the content of the feedback provided. If feedback content maps directly onto an outcome measure, then that measure shall be chosen as the outcome to be included. If feedback does not directly map onto an outcome measure, then prioritization will be as follows: 1) Absolute error, 2) RMSE, 3) Absolute constant error, 4) Absolute timing error, 5) Relative timing error, 6) Variable error, 7), Movement time, 8) Movement form – expert raters, 9) Otherwise unspecified objective performance measure reported first in research report. 9. Data Extraction The data of interest are the standardized mean differences (Hedges’ g) between 100% feedback and reduced feedback frequency groups at a given time-point. To facilitate reproduction, effect sizes will be calculated based on the following priority list: group means and standard deviations, means and standard errors, F values and degrees of freedom, p-values and degrees of freedom. For experiments that included prognostic covariates in the analysis, F-values, degrees of freedom, number of covariates, and correlation between covariates and dependant variables will be used. If the article containing a given experiment does not provide adequate data, an email will be sent to the corresponding author requesting the necessary data and a two-week waiting period will be applied. If data cannot be produced to accurately calculate the effect size for a given experiment that experiment will be left out of further analysis. If full data are produced by the authors and an analysis of covariance can be conducted including pre-test performance as a prognostic covariate, but the original experiment did not perform and analysis of covariance, then both unadjusted and adjusted estimates will be calculated. The unadjusted estimate will be included in the primary meta-analysis and while the adjusted estimates will be included in a subsequent sensitivity analysis. If experiments include multiple groups with reduced feedback frequency, then the group receiving the least frequent (non-zero) feedback will be selected for inclusion in the primary analysis. If reduced frequency groups differ with respect to fading of schedule, the group with the least frequent feedback schedule that is consistent (i.e., not faded) will be included in the main analysis. To be included in the primary meta-analyses, the comparison group cannot receive feedback according to a bandwidth. Additional reduced frequency groups will be included in the moderator analyses of feedback frequency, faded scheduling, and bandwidth scheduling. If experiments include multiple 100% feedback frequency groups, then those groups will be collapsed using the following formula (https://handbook-5-1.cochrane.org/chapter_7/table_7_7_a_formulae_for_combining_groups.htm). In factorial experiments with a significant interaction, the simple effects of the most relevant condition will be included. For example, an experiment may cross neural stimulation with feedback frequency – if there is a significant interaction only the data from the sham conditions will be included. If both levels of the secondary IV are equally relevant but not included as a potential moderator in the meta-analysis, then the main effect will be included even if there is a significant interaction. If the second IV in a factorial experiment is included as a moderator in this meta-analysis (for example, age of participants), then one effect from each level of the IV will be included (for example, one for child, one for adult). If a factorial experiment crosses two IVs that both manipulate feedback frequency (for example, 100% vs.50% KR about absolute timing, 100% vs. 50% KR about relative timing), then the main effect of each IV will be included for the outcome corresponding to the feedback provided (in this example, absolute and relative timing, respectively). Two researchers will independently extract the required data for all time points of interest. Conflicts will be resolved by a third researcher. The compute.es package in R will be used to calculate all effect sizes. 10. Quality Assessment Each article will be assessed using the Cochrane Risk of Bias checklist in the Covidence Review Software. 11. Screening data for influential cases The meta-analysis package ‘metafor’ in R will be used to screen and analyze the data. In order to identify outlier values and exclude them from the final random effects model, the following influence statistics will be calculated for both immediate and delayed retention data: externally standardized residuals, DFFITS values, Cook's distances, covariance ratios, DFBETAS values, the estimates of t2 squared when each study is removed in turn, the test statistics for (residual) heterogeneity when each study is removed in turn, the diagonal elements of the hat matrix, and the weights (in %) given to the observed outcomes during the model fitting. If any study is identified as extremely influential by any three of the influence metrics it will be removed from the primary analysis as an outlier. 12. Data Analysis Alpha will be set at p = .05 for all analyses. To facilitate a decision about equivalence, an effect size of g = .1 will be considered the minimum effect size of interest. A two-one-sided-test (TOST) of equivalence will be conducted on any non-significant result. Publication and other selection biases are expected in this literature. Therefore, the adjusted estimate produced by a weight-function model with a .025 (one-sided) cut-point will be considered the primary outcome in this meta-analysis for all single time point analyses. Since there are currently no bias correction methods derived to account for selection effects on a Time X Group interaction, mixed model analyses will be considered the primary outcomes for these analyses. However, interpretation of these results will depend heavily on the extent of publication bias revealed in the univariate weight-function models. If selection effects are estimated to be substantial and influential for any time point of interest, then conclusions regarding Time X Group interactions will be withheld. The following script will be applied to the dataset of effects sizes with any outliers already removed. Additional rows may need to be removed from the dataset because they are intended for moderator analyses only. This information will be added to the analysis script before final publication. Any deviations from this script will be noted in the final publication and a justification will be provided. # import datasets #terminal feedback frequency data Data # Kantak &amp; Winstein (2012) review data KWdat # Data including bandwidth protocols BandData # load the necessary libraries library(metafor) library(meta) library(weightr) # in order to estimate the average effect of reduced feedback at delayed retention, a naïve random #effects model will be fit to only the experiments that included a delayed retention test #select only delayed retention results delay &lt;- data[!(data$time== “3”),] delayed &lt;- rma(ret_g, ret_v, data = delay) #select only immediate retention results imd &lt;- data[!(data$time== “2”),] # estimate the effect at immediate retention immediate &lt;- rma(ret_g, ret_v, data = imd) #select only acquisition results acq &lt;- data[!(data$time== “1”),] # estimate the effect at immediate retention acquisition &lt;- rma(ret_g, ret_v, data = acq) # test the time (immediate, delayed) X feedback frequency interaction with mixed effects model # dummy code the immediate and delayed levels of the time variable data$t.immediate &lt;- ifelse(data$time == "Immediate", 1, 0) data$t.delay &lt;- ifelse(data$time == "Delayed", 1, 0) Interaction &lt;- rma.mv(ret_g, ret_v, mods = cbind(t.immediate, t.delay), random = cbind(t.immediate, t.delay) | as.factor(study),struct = "CS", data = data) # estimate the effect at immediate and delayed retention while accounting for repeated measures Estimates &lt;- Interaction &lt;- rma.mv(ret_g, ret_v, mods = cbind(t.immediate, t.delay) + intercept = FALSE, random = cbind(t.immediate, t.delay) | as.factor(study),struct = "CS", data = data) # test the time (acquisition, delayed) X feedback frequency interaction with mixed effects model # dummy code the acquisition level of the time variable data$t.acquisition &lt;- ifelse(data$time == "Acquisition", 1, 0) AcqInteraction &lt;- rma.mv(ret_g, ret_v, mods = cbind(t.acquisition, t.delay), random = cbind(t.acquisition, t.delay) | as.factor(study),struct = "CS", data = data) # estimate the effect at acqusition and delayed retention while accounting for repeated measures AcqEstimates &lt;- Interaction &lt;- rma.mv(ret_g, ret_v, mods = cbind(t.acquisition, t.delay) + intercept = FALSE, random = cbind(t.acquisition, t.delay) | as.factor(study),struct = "CS", data = data) # inspect results of each analysis Summary(delayed) Summary(immediate) Summary(acquisition) Summary(Interaction) Summary(Estimates) Summary(AcqInteraction) Aummary(AcqEstimates) # evaluate the influence of publication bias on the delayed retention results x &lt;- delay$ret_g v &lt;- delay$ret_v weightfunct(x,v) # evaluate the influence of publication bias on immediate retention results g &lt;- imd$ret_g s &lt;- imd$ret_v weightfunct(g,s) # evaluate the influence of publication bias on the results reviewed by Kantak &amp; Winstein (2012) #remove immediate retention results KWdelay &lt;- KWdat[!(KWdat$time== “1”),] # fit weight-function model k &lt;- KWdelay$ret_g w &lt;- KWdelay$ret_v weightfunct(k,w) #remove delayed retention results KWimd &lt;- KWdat[!(KWdat$time== “2”),] #fit weight-function model d &lt;- KWimd$ret_g r &lt;- KWimd$ret_v Weightfunct(d,r) # exploratory tests of categorical moderators for delayed retention results # any significant moderator will be followed up with analysis to estimate effect at each level using following code where X = factor heading rma(ret_g,ret_v, mods = ~factor(X)-1, data= delay) # age group rma(ret_g,ret_v, mods = ~factor(age), data= delay) # skill level rma(ret_g,ret_v, mods = ~factor(skill), data= delay) # task class rma(ret_g,ret_v, mods = ~factor(task), data= delay) # publication status rma(ret_g,ret_v, mods = ~factor(pub), data= delay) # faded schedule rma(ret_g,ret_v, mods = ~factor(faded), data= delay) # bandwidth protocols Banddelay &lt;- BandData[!(BandData$time== “3”),] rma(ret_g,ret_v, mods = ~factor(bandwidth), data= Banddelay) # exploratory tests of continuous moderators for delayed retention results # number of acquisition trials rma(ret_g,ret_v, mods = ~ trials, data= delay) # number of acquisition days rma(ret_g,ret_v, mods = ~ days, data= delay) # frequency of terminal feedback rma(ret_g,ret_v, mods = ~ frequency, data= delay) # exploratory tests of categorical moderators for immediate retention results # age group rma(ret_g,ret_v, mods = ~factor(age), data= imd) # skill level rma(ret_g,ret_v, mods = ~factor(skill), data= imd) # task class rma(ret_g,ret_v, mods = ~factor(task), data= imd) # publication status rma(ret_g,ret_v, mods = ~factor(pub), data= imd) # exploratory tests of continuous moderators for immediate retention results # number of acquisition trials rma(ret_g,ret_v, mods = ~ trials, data= imd) # number of acquisition days rma(ret_g,ret_v, mods = ~ days, data= imd) # frequency of terminal feedback rma(ret_g,ret_v, mods = ~ frequency, data= imd) 12. Sensitivity analyses Sensitivity analyses will be conducted on the Time X Feedback frequency interaction analyses as well as on the publication bias analysis. A) Standardized mean change will be calculated wherever data allow to test the Time X Feedback frequency interaction. Data required: Immediate retention mean and standard deviation, delayed retention mean and standard deviation, sample size, and correlation between immediate and delayed retention. The correlation between time-points is expected to be unknown in the majority of cases. Therefore, any known correlations will be used to estimate a range of plausible values. Sensitivity analyses will be conducted using the Standardized Mean Change with a correlation value estimated from each quartile of the overall range of correlation estimates. Therefore, there will be three runs of this analysis. The analysis will be coded and run according to the methods described here: http://www.metafor-project.org/doku.php/analyses:morris2008 B) Publication bias correction methods are sensitive to unknown underlying conditions. Therefore, on the basis of the first run of analyses, performance checks under a range of plausible conditions will be run using the following application: http://www.shinyapps.org/apps/metaExplorer/ On the basis of these performance checks, additional bias correction methods will be applied in an attempt to achieve the most possible coverage of plausible conditions. One additional need for sensitivity analysis may emerge following our quality assessment of included articles. Analyses may be re-run with articles judged to have a high risk of bias excluded in order to evaluate the extent that the overall analysis is sensitive to the inclusion of high risk studies.