Package: evident
EvidentEvident is a tool for performing effect size and power calculations on microbiome data.InstallationYou can install the most up-to-date version of Evident from PyPi using the following command:pipinstallevidentQIIME 2Evident is also available as aQIIME 2plugin.
Make sure you have activated a QIIME 2 environment and run the same installation command as above.To check that Evident installed correctly, run the following from the command line:qiimeevident--helpYou should see something like this if Evident installed correctly:Usage:qiimeevident[OPTIONS]COMMAND[ARGS]...Description:Performpoweranalysisonmicrobiomedata.Supportscalculationofeffectsizegivenmetadatacovariatesandsupportingvisualizations.Pluginwebsite:https://github.com/biocore/evidentGettingusersupport:PleaseposttotheQIIME2forumforhelpwiththisplugin:https://forum.qiime2.org

Options:--versionShowtheversionandexit.--example-dataPATHWriteexampledataandexit.--citationsShowcitationsandexit.--helpShowthismessageandexit.

Commands:multivariate-effect-size-by-categoryMultivariatedataeffectsizebycategory.multivariate-power-analysisMultivariatedatapoweranalysis.plot-power-curvePlotpowercurve.univariate-effect-size-by-categoryUnivariatedataeffectsizebycategory.univariate-power-analysisUnivariatedatapoweranalysis.univariate-power-analysis-repeated-measuresUnivariatedatapoweranalysisforrepeatedmeasures.visualize-resultsTabulateevidentresults.Standalone UsageEvident can operate on two types of data:Univariate (vector)Multivariate (distance matrix)Univariate data can be alpha diversity. log ratios, PCoA coordinates, etc.
Multivariate data is usually a beta diversity distance matrix.For this tutorial we will be using alpha diversity values, but the commands are nearly the same for beta diversity distance matrices.First, open Python and import EvidentimportevidentNext, load your diversity file and sample metadata.importpandasaspdmetadata=pd.read_table("data/metadata.tsv",sep="\t",index_col=0)faith_pd=metadata["faith_pd"]The main data structure in Evident is the 'DataHandler'.
This is the way that Evident stores the data and metadata for power calculations.
For our alpha diversity example, we'll load theUnivariateDataHandlerclass from Evident.UnivariateDataHandlertakes as input the pandas Series with the diversity values and the pandas DataFrame containing the sample metadata.
By default, Evident will only consider metadata columns with, at max, 5 levels.
To modify this behavior, provide a value for themax_levels_per_categoryargument.
Additionally, Evident will not consider any category levels represented by fewer than 3 samples.
To modify this behavior, use themin_count_per_levelargument.adh=evident.UnivariateDataHandler(faith_pd,metadata)Next, let's say we want to get the effect size of the diversity differences between two groups of samples.
We have in our example a column in the metadata "classification" comparing two groups of patients with Crohn's disease.
First, we'll look at the mean of Faith's PD between these two groups.metadata.groupby("classification").agg(["count","mean","std"])["faith_pd"]which results incount       mean       std
classification
B1                 99  13.566110  3.455625
Non-B1            121   9.758946  3.874911Looks like there's a pretty large difference between these two groups.
What we would like to do now is calculate the effect size of this difference.
Because we are comparing only two groups, we can use Cohen's d.
Evident automatically chooses the correct effect size to calculate - either Cohen's d if there are only two categories or Cohen's f if there are more than 2.adh.calculate_effect_size(column="classification")This tells us that our effect size is 1.03.Now let's say we want to see how many samples we need to be able to detect this difference with a power of 0.8.
Evident allows you to easily specify arguments for alpha, power, or total observations for power analysis.
We can then plot these results as a power curve to summarize the data.fromevident.plottingimportplot_power_curveimportnumpyasnpalpha_vals=[0.01,0.05,0.1]obs_vals=np.arange(10,101,step=10)results=adh.power_analysis("classification",alpha=alpha_vals,total_observations=obs_vals)plot_power_curve(results,target_power=0.8,style="alpha",markers=True)When we inspect this plot, we can see how many samples we would need to collect to observe the same effect size at different levels of significance and power.Interactive power curve with BokehEvident allows users tointeractivelyperform effect size and power calculations usingBokeh.
To create a Bokeh app, use the following command:fromevident.interactiveimportcreate_bokeh_appcreate_bokeh_app(adh,"app")This will save the necessary files into a new directoryapp/.
Navigate to the directory containingapp/(notapp/itself) and execute this command from your terminal:bokehserve--showappThis should open up a browser window with the interactive visualizations.
The "Summary" tab gives an overview of the data and the effect sizes/power.
Barplots showing the metadata effect sizes for both binary and multi-class categories (ranked in descending order) are shown.
On the right is a dynamic power curve showing the power analysis for metadata columns.
The significance level, total observation range, and chosen columns can be modified by using the control panel on the left side of the tab.Swap to the "Data" tab using the bar on the top.
Here you can see boxplots of the data for each metadata category.
Select a column from the dropdown to change which data is shown.
You can also check the "Show scatter points" box to overlay the raw data onto the boxplots.Note that because evident uses Python to perform the power calculations, it is at the momentnotpossible to embed this interactive app into a standalone webpage.QIIME 2 UsageEvident provides support for the popular QIIME 2 framework of microbiome data analysis.
We assume in this tutorial that you are familiar with using QIIME 2 on the command line.
If not, we recommend you read the excellentdocumentationbefore you get started with Evident.
Note that we have only tested Evident on QIIME 2 version 2021.11.
If you are using a different version and encounter an error please let us know via an issue.To calculate power, we can run the following command:qiimeevidentunivariate-power-analysis\--m-sample-metadata-filemetadata.qza\--m-sample-metadata-filefaith_pd.qza\--p-data-columnfaith_pd\--p-group-columnclassification\--p-alpha0.010.050.1\--p-total-observations$(seq1010100)\--o-power-analysis-resultsresults.qzaWe provide multiple sample metadata files to QIIME 2 because they are internally merged.
You should provide a value for--p-data-columnso Evident knows which column in the merged metadata contains the numeric values (this is only necessary for univariate analysis).
In this case, the name of thefaith_pd.qzavector isfaith_pdso we use that as input.Notice how we used$(seq 10 10 100)to provide input into the--p-total-observationsargument.seqis a command on UNIX-like systems that generates a sequence of numbers.
In our example, we usedseqto generate the values from 10 to 100 in intervals of 10 (10, 20, ..., 100).With this results artifact, we can visualize the power curve to get a sense of how power varies with number of observations and significance level.
Run the following command:qiimeevidentplot-power-curve\--i-power-analysis-resultsresults.qza\--p-target-power0.8\--p-stylealpha\--o-visualizationpower_curve.qzvYou can view this visualization atview.qiime2.orgdirectly in your browser.ParallelizationEvident provides support for parallelizing effect size calculations throughjoblib.
Parallelization is performed across different columns when usingeffect_size_by_categoryandpairwise_effect_size_by_category.
Consider parallelization if you have a lot of samples and/or a lot of different metadata categories of interest.
By default, no parallelization is performed.With Python:fromevident.effect_sizeimporteffect_size_by_categoryeffect_size_by_category(adh,["classification","cd_resection","cd_behavior"],n_jobs=2)With QIIME 2:qiimeevidentunivariate-effect-size-by-category\--m-sample-metadata-filemetadata.qza\--m-sample-metadata-filefaith_pd.qza\--p-data-columnfaith_pd\--p-group-columnsclassificationsexcd_behavior\--p-n-jobs2\--o-effect-size-resultsalpha_effect_sizes.qzaRepeated MeasuresEvident supports limited analysis of repeated measures.
When your dataset has repeated measures, you can calculateeta_squaredfor univariate data.
Note that multivariate data is not supported for repeated measures analysis.
Power analysis for repeated measures implements a repeated measures ANOVA.
Additionally, when performing power analysisonlypower can be calculated (in contrast toUnivariateDataHandlerandMultivariateDataHandlerwhere alpha, significance, and observations can be calculated).
This power analysis assumes that the number of measurements per group is equal.With Python:fromevident.data_handlerimportRepeatedMeasuresUnivariateDataHandlerrmadh=RepeatedMeasuresUnivariateDataHandler(faith_pd,metadata,individual_id_column="subject",)effect_size_result=rmadh.calculate_effect_size(state_column="group")power_analysis_result=rmadh.power_analysis(state_column="group",subjects=[2,4,5],measurements=10,alpha=0.05,correlation=[-0.5,0,0.5],epsilon=0.1)With QIIME 2:qiime evident univariate-power-analysis-repeated-measures \
    --m-sample-metadata-file metadata.qza \
    --m-sample-metadata-file faith_pd.qza \
    --p-data-column faith_pd \
    --p-individual-id-column subject \
    --p-state-column group \
    --p-subjects 2 4 5 \
    --p-measurements 10 \
    --p-alpha 0.05 \
    --p-correlation -0.5 0 0.5 \
    --p-epsilon 0.1Help with EvidentIf you encounter a bug in Evident, please post a GitHub issue and we will get to it as soon as we can.
We welcome any ideas or documentation updates/fixes so please submit an issue and/or a pull request if you have thoughts on making Evident better.If your question is regarding the QIIME 2 version of Evident, consider posting to theQIIME 2 forum.
You can open an issue on theCommunity Plugin Supportboard and tag@gibsramenif required.