Using the TIDieR Checklist to Standardize the Description of a Functional Strength Training Intervention for the Upper Limb After Stroke

<jats:sec>             <jats:title>Background and Purpose:</jats:title>             <jats:p>Published reports of intervention in randomized controlled trials are often poorly described. The Template for Intervention Description and Replication (TIDieR) checklist has been recently developed to improve the reporting of interventions. The aim of this article is to describe a therapy intervention used in the stroke rehabilitation trial, “Clinical Efficacy of Functional Strength Training for Upper Limb Motor Recovery Early After Stroke: Neural Correlates and Prognostic Indicators” (FAST-INdICATE), using TIDieR.</jats:p>           </jats:sec>           <jats:sec>             <jats:title>Methods:</jats:title>             <jats:p>The functional strength training intervention used in the FAST-INdICATE trial was described using TIDieR so that intervention can be replicated by both clinicians, who may implement it in practice, and researchers, who may deliver it in future research. The usefulness of TIDieR in the context of a complex stroke rehabilitation intervention was then discussed.</jats:p>           </jats:sec>           <jats:sec>             <jats:title>Results and Discussion:</jats:title>             <jats:p>The TIDieR checklist provided a systematic way of describing a treatment intervention used in a clinical trial of stroke rehabilitation. Clarification is needed regarding several aspects of the TIDieR checklist, including in which section to report about the development of the intervention in pilot studies, results of feasibility studies; overlap between training and procedures for assessing fidelity; and where to publish supplementary material so that it remains in the public domain.</jats:p>           </jats:sec>           <jats:sec>             <jats:title>Conclusions:</jats:title>             <jats:p>TIDieR is a systematic way of reporting the intervention delivered in a clinical trial of a complex intervention such as stroke rehabilitation. This approach may also have value for standardizing intervention in clinical practice.</jats:p>             <jats:p>                                  <jats:bold>Video abstract available</jats:bold> for more insights from the authors (see Supplemental Digital Content 1, http://links.lww.com/JNPT/A131).</jats:p>           </jats:sec>

section_content: i) Organize your data to comply with the FAIR Data Principles (https://www. gofair. org/fair-principles/). This makes it easier to access and reuse by others. ii) We recommend archiving data/code in a version-controlled repository that provides a permanent digital object identifier (DOI) for code/data, making them citable:

(1) Open Science Framework (OSF; https://osf. io/);

(2) Dryad (https://datadryad. org/stash);

(3) Zenodo (https://zenodo. org/); (4) See also the more exhaustive list of recommendations from the journal Scientific Data: https://www. nature. com/sdata/policies/repositories 4) We strongly recommend that authors follow existing reporting guidelines for different study designs. This includes CONSORT for randomized trials, STROBE for observational studies, and PRISMA for systematic reviews and meta-analyses. Authors may download these guidelines at the Equator Network: https://www. equatornetwork. org/). 

5) Authors should clearly distinguish between prespecified analyses and exploratory/post-hoc analyses. Exploratory/post-hoc analyses have a high chance of producing spurious findings; thus, interpretations of these analyses should be appropriately cautious. a) We generally encourage authors to preregister their analyses, as one of the benefits of preregistration is that it allows stakeholders to clearly distinguish between prespecified and post-hoc analyses (https://link. springer. com/article/10. 1007/s40279-019-01227-1). b) Note that for some types of studies, preregistration is mandatory, such as clinical trials (e. g., clinicaltrials. gov) or systematic reviews and meta-analyses (e. g., the PROSPERO registry). c) Outside of formal registries and for any study design (e. g., secondary data analysis), authors are encouraged to use supplemental tools to register their analysis plan (e. g., OSF, AsPredicted. org) 6) Authors should clearly provide an explanation for how the sample size was determined. For some studies, this may come in the form of an a priori power analysis detailing a specific number of participants to be recruited. For others, it may require detailing logistical constraints (e. g., studying rare events, cost prohibitive methods, etc. ). It is inappropriate to use data from the current sample to calculate post-hoc power (see explanation below). a) For prespecified analyses, authors should justify the adequacy of their sample size in the Methods section by reporting the results of a priori sample size calculation(s) for the main statistical test(s) of the primary outcome(s) (https://pubmed. ncbi. nlm. nih. gov/32814615/, https://pubmed. ncbi. nlm. nih. gov/32844536/). b) If an a priori sample size calculation was not performed (as in the case of a secondary data analysis) and authors would like to demonstrate that their sample size was adequate for their analytic aims, they may provide a power calculation based on a minimum effect size of interest (https://doi. org/10. 1525/collabra. 33267). 

c) Sample size calculations may either be based on ensuring sufficient power for testing a specific hypothesis test or on ensuring sufficient precision for estimating an effect (e. g., in measurement studies, it is more important to estimate the ICC with precision, rather than show the ICC is statistically different from 0). d) Power/sample size calculations must include sufficient information to replicate the calculation, including the alpha level (for hypothesis testing), effect size assumed, variability assumed, and the specific statistical test conducted. e) Authors should not report "post-hoc power" calculations based on the observed effect size. Post hoc power assumes that the observed effect size is the true effect size. (e. g., p<. 05). P-values are one piece of information that needs to be discussed in balance with effect size, uncertainty, and the study context. ii) P-values greater than the specified alpha, e. g., p>. 05, should not be interpreted as evidence of "no effect" or "no difference. "

(1) P>0. 05 should be interpreted as an "equivocal" finding. That is, when P>α there is insufficient evidence to reject the null hypothesis, and the absence of evidence is not evidence of absence. 

(https://pubmed. ncbi. nlm. nih. gov/7647644/) iii) Report precise p-values, e. g., p=. 002, rather than ranges such as p<. 05 or p<. 01. 

(1) As most studies at JNPT use α=0. 05, p-values in the text should be reported to at least 3 decimals (e. g., p=0. 051 versus 0. 049). 

(2) Very small p-values, e. g., p=0. 00008 may be reported as p<. 001 or use scientific notation p=8E-5. iv) Don't report p-values in isolation, particularly in the abstract of the paper. 

Always try to include an effect size, confidence interval, regression coefficient, etc., alongside the corresponding p-value. v) P-values should not be interpreted as giving the probability of a hypothesis, e. g., p=. 05 does not indicate that there is a 5% chance that the null hypothesis is true. 

(1) E. g., the best explanation of the data might be that it's a false positive even though p<0. 05 (see https://storkjournals. org/index. php/cik/article/view/49/78) 2) Authors should not report "here not there" statistical significance between groups. For example, "The improvement was significant in the treatment group (p=. 02) but not in the control group (p=. 43 a) The 95% level of confidence/credibility is a reasonable standard, though authors may use other levels if justified. b) To assist with correct interpretations of the confidence interval, recall that confidence intervals show the range of possible true effects that are compatible with data (https://link. springer. com/article/10. 1186/s12874-020-01105-9). When interpreting a confidence interval, authors should discuss the implications of all values in that range. c) Thus, authors should *not* interpret confidence intervals that barely contain the null value as evidence of an effect. These kinds of claims are about the probability of a hypothesis given the data [p(H0|data)] when the CI and the p-value reflect the probability of the data given a specific hypothesis [p(data| H0)]. These probabilities are not the same [p(data| H0) =/= p(H0|data)] and to accurately translate between them requires Bayesian statistics with a wellspecified prior distribution. 6) When reporting effect sizes and confidence intervals, "raw" effect sizes are often preferable to standardized effect sizes. a) For instance, when comparing the difference in gait speed between two groups, it is more clinically meaningful to report the difference in meters per second (m/s) rather than in units of standard deviations (as in Cohen's d or Hedges' g). b) Similarly, if reporting a regression slope for the association between physical activity and age, it might be more meaningful to report the change in physical activity per year ([hr/d]/y) than the variance explained by the effect of age (r^2). c) However, note that for many psychological instruments, there are no inherently meaningful units, and therefore reporting standardized effect sizes might be preferred. d) Similarly, there are also times when it is actually preferable to report standardized effect sizes like partial-eta^2 (from factorial ANOVA) or the change in r^2 (from a multi-variable regression), such as when reporting a test with multiple degrees of freedom or trying to assess the fit of a model 7) Authors should not interpret p-values close to their chosen α-level (typically α=0. 05) as "marginally significant", "nearing significance", "evidence of a trend", or use any other similar language. a) When using a hypothesis testing framework, results are either statistically significant or they are not. b) However, see also the points above that p>0. 05 is not evidence of no difference and a p-value is only one piece of information authors should use to interpret their results. c) Authors are welcome to interpret p-values continuously, but need to understand that this compromises Type 1 error control. E. g., p = 0. 06 reflects data just about as unlikely under the null as p = 0. 05 or p=0. 04. However, if authors decide to reject the null when p=0. 06, then they should have really set α=0. 06 or higher. Instead, authors are welcome to interpret the p-value continuously without reference to "statistical significance" or "rejecting the null hypothesis". In those cases, though authors will have to defend their choice of where an effect becomes statistically interesting/meaningful and defend the potential increase in the Type 1 error rate to reviewers. i) Furthermore, if authors do choose to interpret p-values continuously, we recommend they supplement p-values with "surprise values" (or s-values). The s-value if the -log2(p-value) and can be re-interpreted "as the number of heads/tails in a row on a fair coin flip". 

(1) For instance -log2(0. 05) = 4. 32, so getting p=0. 05 is about as surprising as getting 4-heads in a row, which happens a lot. (2) Whereas, -log2(0. 001) = 9. 97, so getting p=0. 001 is about as surprising as getting 10-heads in a row, which almost never happens on a fair coin. Provided that the mechanism of missingness is incorporated into the model, many imputation/analysis methods produce unbiased results under MAR. iii) MNAR = data that are neither MCAR or MAR. MNAR could be because we cannot identify/account for the mechanisms of missingness (otherwise we could treat it as MAR). Or MNAR could be because the value of the missing variable is the reason it is missing. E. g., if "depression" is our outcome measure, we might be missing data from severely depressed people specifically because they have severe depression. MNAR creates bias for most methods of imputation and analysis (e. g., we cannot confidently impute data for people with severe depression from people with mild depression). iv) Note that data seldom contain all of the information to prove that data are either confidently completely MAR or MNAR. Thus, authors and reviewers should rely on a rational analysis of the data and the methods to determine the influence that missing data might have on the results. However, at least some discussion of missing data will always be warranted when missing data are present.

section_content: Authors are increasingly using machine learning and other types of predictive modeling approaches. Below we present a few general guidelines for describing a machine learning workflow and the presentation of the results. Machine learning algorithms are far too numerous to address the sufficient reporting of all possible unsupervised, supervised, or reinforcement learning methods. However, we encourage authors/reviewers to consult these accessible references: https://pubmed. ncbi. nlm. nih. gov/31007185/, https://www. cs. mcgill. ca/~jpineau/ReproducibilityChecklist. pdf, and https://www. ahajournals. org/doi/10. 1161/CIRCOUTCOMES. 120. 006556). 

1) Authors should clearly explain how the algorithms chosen align with the research question. For instance, unsupervised learning algorithms are hypothesisfree approaches to understanding the underlying dimensions of high dimensional data (e. g., PCA) or how individuals cluster together in high dimensional spaces (e. g., k-means cluster analysis). In contrast, supervised learning algorithms provide powerful tools for predictive modeling that may be less dependent on distributional assumptions or better at finding high-dimensional, non-linear patterns compared to traditional regression methods. 

2) The ML analysis pipeline needs to be fully described, from data cleaning procedures to model evaluation. Every ML model should be described with sufficient detail that the following 5 areas are explained: a) Data sources, data cleaning, and any other methods of pre-processing. i) Given the large sample sizes required for many ML analyses, manual inspection of the data is often not possible. Therefore, the authors should explain all steps for automated quality control and be transparent about any limitations to quality assurance. b) Feature selection i) Note, unsupervised learning algorithms may often be used for reducing highdimensional data into the smallest number of the most informative features. ii) Similarly, supervised/reinforcement learning algorithms can also be used to determine relative levels of variable importance, suggesting which features could be retained in subsequent models. c) Model training i) Any cross-validation needs to be explained in detail. 

ii) All hyper-parameters for an algorithm need to be stated and the values used for those hyper parameters need to be defended (e. g., LASSO is an algorithm that adds a regularization hyperparameter to ordinary least squares regression, the chosen value of this hyper parameter affects the degree of shrinkage and therefor the results). d) Testing and evaluation b) The standard error quantifies the uncertainty in a statistic (e. g., mean height) and not the variability in the trait itself (e. g., height). c) The standard error confounds variability (σ) with sample size (N), so authors are encouraged to report/plot a confidence interval, when estimation is the goal, or report/plot the standard deviation, when description of the sample is the goal. 4) When reporting descriptive statistics for variables with highly skewed distributions, authors should report the median and interquartile range instead of (or in addition to) the mean and standard deviation. 5) Tables and figures should □ 15. In the reviewer's opinion, the authors' conclusions are justified by the data?

(Please list any problematic cases. )
section_name: C. Machine Learning (ML) / Artificial Intelligence (AI) Modeling Approaches

section_content: i) Predictive models should be evaluated in an independent dataset that was absolutely not included in the training dataset. ii) Note, authors should only use the term "prediction" to describe their results when the model is validated in this independent dataset. iii) If authors have a predictive model but only do internal validation on the training data (e. g., k-fold cross validation with logistic regression) then the term "explanation" should be used instead of "prediction". 

Finally, many concepts in machine learning have an analogous concept in applied statistics. As many authors/reviewers might be more familiar with the statistical names, we provide a table to help translate between similar concepts here: