Package: tarpan
A Python library for analysing cmdstanpy outputThis is a collection of functions for analysing output ofcmdstanpylibrary. The main idea is to do a quick data analysis by calling a single function that makes:traceplots of samples,text and plots of the summaries of model parameters,histograms and pair plots of posterior distributions of parameters.The only known illustration of a tarpan made from life, depicting a five month old colt (Borisov, 1841). Source:Wikimedia Commons.SetupFirst, run:pip install tarpanFinally, installcmdstanby running:install_cmdstan -v 2.21.0Complete analysis:save_analysisThis is the main function of the library that saves summaries
and trace/pair/tree plots inmodel_infodirectory.
The function is useful when you want to generate all types of summaries
and plots at once.fromtarpan.cmdstanpy.analyseimportsave_analysismodel=CmdStanModel(stan_file="your_model.stan")fit=model.sample(data=your_data)save_analysis(fit,param_names=['mu','sigma'])Full example codeIf you don't need everything, you can call individual
functions described below to make just one type of plot or a summary.Analysis without cmdstanpyHere is how to analyse values from Pandas' data frame columns:fromtarpan.shared.analyseimportsave_analysissave_analysis(df,param_names=['mu','sigma'])Full example codeSummary:save_summaryCreates a summary of parameter distributions and saves it in text and CSV files.fromtarpan.cmdstanpy.summaryimportsave_summarymodel=CmdStanModel(stan_file="your_model.stan")fit=model.sample(data=your_data)save_summary(fit,param_names=['mu','tau','eta.1'])Full example codeCallprint_summaryto print summary to console instead of saving to a file.The text summary format is such that the text can be pasted into Github/Gitlab/Bitbucket's Markdown file, like this:NameMeanStdMode+-68CI-68CI+95CI-95CI+N_EffR_hatmu8.055.127.534.634.592.9312.16-1.8418.7415401.00tau6.415.722.365.412.350.007.760.0017.0711751.00eta.10.390.920.600.711.13-0.531.31-1.482.1935051.00Summary columnsName, Mean, Stdare the name of the parameter, its mean and standard deviation.68CI-, 68CI+, 95CI-, 95CI+are the 68% and 95% HPDIs (highest posterior density intervals). These valuesare configurable.Mode, +, -is a mode of distribution with upper and lower uncertainties, which are calculated as distances to 68% HPDI.N_Effis Stan's number of effective samples, the higher the better.R_hatis a Stan's parameter representing the quality of the sampling. This value needs to be around 1.00. After generating a model I usually immediately look at this R_hat column to see if the sampling was good.Summary without cmdstanpyHere is how to make summary of values from Pandas' data frame columns:fromtarpan.shared.summaryimportsave_summarysave_summary(df,param_names=['mu','sigma'])Full example codeTree plot:save_tree_plotThis function shows exactly the same information assave_summary, but in
the form a plot. The markers are the modes of the distributions, and the two error bars
indicate 68% and 95% HPDIs (highest posterior density intervals).fromtarpan.cmdstanpy.tree_plotimportsave_tree_plotmodel=CmdStanModel(stan_file="your_model.stan")fit=model.sample(data=your_data)save_tree_plot([fit],param_names=['mu','sigma'])Full example codeTree plot without cmdstanpyOne can make a tree plot by supplying a Panda's data frame that shows summaries of values for the frame's columns:fromtarpan.shared.tree_plotimportsave_tree_plotsave_tree_plot([df],param_names=['mu','sigma'])Full example codeComparing multiple models on a tree plotSupply multiple fits in order to compare parameters from multiple models.fromtarpan.cmdstanpy.tree_plotimportsave_tree_plotfromtarpan.shared.tree_plotimportTreePlotParams# Sample from two modelsmodel1=CmdStanModel(stan_file="your_model1.stan")fit1=model1.sample(data=your_data)model2=CmdStanModel(stan_file="your_model2.stan")fit2=model2.sample(data=your_data)# Supply legend labels (optional)tree_params=TreePlotParams()tree_params.labels=["Model 1","Model 2","Exact"]data=[{"mu":2.2,"tau":1.3}]# Add extra markers (optional)save_tree_plot([fit1,fit2],extra_values=data,param_names=['mu','tau'],tree_params=tree_params)Full example codeComparing parameters of multiple modelsUsesave_compare_parametersfunction to compare parameters between different models:mutauModel 17.53 (+4.63, -4.59)2.36 (+5.41, -2.35)Model 28.87 (+9.05, -9.50)3.64 (+8.14, -3.61)This table is a numerical version of the plot created bysave_tree_plot. The values here are modes of the distributions and uncertainties are distances to 68% HPD intervals.fromtarpan.cmdstanpy.compare_parametersimportsave_compare_parametersextra=[{"mu":2.2,"theta":1.3}]# Add extra values (optional)save_compare_parameters([fit1,fit2],labels=['Model 1','Model 2','Extra'],extra_values=extra,param_names=["mu","theta"])Full example codeUsesave_compare_parameterswithout cmdstanpyHere is how to compare parameters using Pandas data frames df1 and df2:fromtarpan.shared.compare_parametersimportsave_compare_parametersextra=[{"mu":2.2,"theta":1.3}]# Add extra values (optional)save_compare_parameters([df1,df2],labels=['Model 1','Model 2','Extra'],extra_values=extra,param_names=["mu","theta"])Full example codeTrace plot:save_traceplotThe plot shows the values of parameters samples.
Different colors correspond to samples form different chains. Ideally,
the lines of different colors on the left plots are well mixed, and the
right plot is fairly uniform.fromtarpan.cmdstanpy.traceplotimportsave_traceplotmodel=CmdStanModel(stan_file="your_model.stan")fit=model.sample(data=your_data)save_traceplot(fit,param_names=['mu','tau','eta.1'])Full example codePair plot:save_pair_plotThe plot helps to see correlations between parameters and spot funnel
shaped distributions that can result in sampling problems.fromtarpan.cmdstanpy.pair_plotimportsave_pair_plotmodel=CmdStanModel(stan_file="your_model.stan")fit=model.sample(data=your_data)save_pair_plot(fit,param_names=['mu','tau','eta.1'])Full example codePair plot without cmdstanpyHere is how to make a pair plot of values from Pandas' data frame columns:fromtarpan.shared.pair_plotimportsave_pair_plotsave_pair_plot(df,param_names=['mu','sigma'])Full example codeHistogram:save_histogramShow histograms of parameter distributions.fromtarpan.cmdstanpy.histogramimportsave_histogrammodel=CmdStanModel(stan_file="your_model.stan")fit=model.sample(data=your_data)save_histogram(fit,param_names=['mu','tau','eta.1','theta.1'])Full example codeHistogram without cmdstanpyHere is how to make histograms of values from Pandas' data frame columns:fromtarpan.shared.histogramimportsave_histogramsave_histogram(df,param_names=['mu','sigma'])Full example codeComparing modelsRunsave_compareto compare multiple models using WAIC and PSIS methods in order to see which models are more compatible with the data.fromtarpan.cmdstanpy.compareimportsave_comparemodel1=CmdStanModel(stan_file="your_model1.stan")fit1=model1.sample(data=your_data)model2=CmdStanModel(stan_file="your_model2.stan")fit2=model2.sample(data=your_data)models={"Model":fit1,"Another model":fit2}save_compare(models=models,lpd_column_name="lpd_pointwise")Here thelpd_column_nameparameter takes the name of the array variable from thegenerated quantitiesblock that contains log probability densities of all data points.Full example codeWAIC and PSIS plotsThesave_comparefunction will create the plots showing WAIC and PSIS values (red round markers). The lower WAIC and PSIS values mean the model is more compatible with the data. The blue triangle marker show the difference between the model and the best model. The error bars correspond to standard errors.Numerical summaries for WAIC and PSISThesave_comparefunction will also create text and csv files which are text versions of the above plots.Summary columns are:PSIS/WAIC, SE: PSIS and WAIC values and their standard errors.dPSIS/dWAIC, dSE: The difference of PSIS and WAIC from the best model (i.e. model with lowest WAIC/PSIS) and the standard error of this difference.pWAIC/pPSIS: the penalty (aka effective number of parameters). The purpose of number is
to combat overfitting. Penalties are already included in the WAIC/PSIS numbers, so models with too many parameters will have larger penalties, and therefore, larger WAIC/PSIS values.Weight: Very approximate measure of the relevance of the model, with higher numbers
correspond to models that are more compatible with the data. Since this number is approximate and does not have uncertainty, it's better to use dWAIC/dPSIS with dSE to compare models.MaxK: The maximum value of Pareto K parameter from the observations. If this value is above 0.5, and especially above 0.7, the PSIS/WAIC model comparisons might not be reliable.WAIC summaryWAICSEdWAICdSEpWAICWeightFungus+treatment361.4813.363.491.00Treatment402.6810.6641.209.822.530.00Intercept405.8811.2944.4011.561.530.00PSIS summaryPSISSEdPSISdSEpPSISMaxKWeightFungus+treatment361.4813.363.490.251.00Treatment402.6910.6741.219.822.540.330.00Intercept405.8811.2944.4011.561.530.280.00Plots of Pareto K valuesThesave_comparefunction creates plots of Pareto K values for  data points. Points with Pareto K values higher than 0.7 are highlighted in red, with their indices shown below the markers. The red points are the ones that have large influence on the model. Having points above 0.7 could mean that WAIC and PSIS are failing and their results should be used with caution and large neon coloured disclaimers.Saving cmdstan samples to diskIt saves a lot of time to sample the model and save the results to disk, so
they can be used on the next run instead of waiting for the sampling again.
This can be done withrunfunction:fromtarpan.cmdstanpy.cacheimportrun# Your function that creates CmdStanModel, runs its `sample` method# and returns the result.## This function must take `output_dir` input parameter and pass it to `sample`.## It may also have any other parameters you wish to pass from `run`.defrun_stan(output_dir,other_param):model=CmdStanModel(stan_file="my_model.stan")fit=model.sample(data=data,output_dir=output_dir# Pass to make CSVs in correct location)returnfit# Return the fit# Will run `run_stan` once, save model to disk and read it on next callsfit=run(func=run_stan,other_param="some data")Full example codeScatter and KDE plotThesave_scatter_and_kdefunction saves a scatter and corresponding KDE (kernel density estimate) plot. The KDE plot takes into account uncertainties of individual values:fromtarpan.plot.kdeimportsave_scatter_and_kdevalues1=[-0.99,-1.37,-1.38,-1.51,-1.29,-1.34,-1.50,-0.93,-0.83,-1.46,-1.07,-1.28,-0.73]uncertainties1=[0.12,0.05,0.11,0.18,0.03,0.19,0.18,0.12,0.19,0.09,0.11,0.16,0.08]values2=[-1.22,-1.15,-0.97,-0.68,-0.37,-0.48,-0.73,-0.61,-1.32,-0.62,-1.13,-0.65,-0.90,-1.29,-1.19,-0.54,-0.64,-0.45,-1.21,-0.75,-0.66,-0.71,-0.61,-0.59,-1.07,-0.65,-0.59]uncertainties2=[0.13,0.14,0.17,0.07,0.11,0.12,0.23,0.05,0.04,0.30,0.11,0.13,0.16,0.03,0.18,0.20,0.16,0.16,0.11,0.09,0.20,0.10,0.08,0.04,0.04,0.23,0.19]save_scatter_and_kde(values=[values1,values2],uncertainties=[uncertainties1,uncertainties2],title="Sodium abundances in RGB stars of NGC 288",xlabel="Sodium abundance [Na/H]",ylabel=["Star number","Probability density"],legend_labels=["AGB","RGB"])gaussian_kdefunctionThe function returns the values for a KDE plot, taking into account uncertainties of individual values:fromtarpan.plot.kdeimportgaussian_kdeimportnumpyasnpimportmatplotlib.pyplotaspltx=np.linspace(0,1,100)y=gaussian_kde(x,values,uncert)plt.fill_between(x,y)Make posterior-scatter-kde plotThesave_posterior_scatter_and_kdefunction makes a scatter-KDE plots
of the data, same assave_scatter_and_kde. In addition, it plots
the posterior distributions.fromtarpan.plot.posteriorimportsave_posterior_scatter_and_kde# Plot one sample from posterior distributiondefmodel_pdf(x,row):mu=row['mu.1']sigma=row['sigma']returnstats.norm.pdf(x,mu,sigma)fig,axes=save_posterior_scatter_and_kde(fits=[fit1,fit2],# Two models returned by model.sample functionpdf=model_pdf,# Function that plot posterior distributionvalues=[data1["y"],data2["y"]],uncertainties=[data1["uncertainties"],data2["uncertainties"]],title="Sodium abundances in RGB stars of NGC 288",xlabel="Sodium abundance [Na/H]",ylabel=["Star number","Probability density"],legend_labels=["AGB","RGB"])Full example codeCommon questionsHow to change the widths of HPD intervals?Where are plot/summary files placed and how to change that?Run unit testspytestThe unlicenseThis work is inpublic domain.🐴🐴🐴This work is dedicated toTarpan, an extinct subspecies of wild horse.Special thanks to Richard McElreath, who wroteStatistical Rethinking textbook, as well asStanandarvizpeople.