Package: hybayes
Bayesian Hypothesis AssessmentA Bayesian hypothesis assessment framework, tailored to comparing pairs of experiments. This package is preliminary
implementation statistical analyses described inthis paper. This can also be seen as a complimentary totestSignificanceNLPpackage which contains popular Frequentist tests.We focus on the setup where the performances of two algorithms, applied on a certain dataset, are provided based on an
evaluation measure. The comparison is done by studying the posterior probabilities (rather than binary commonly-used binary decisions in other tests.)For further discussion of the approaches implemented here (and their comparison to other techniques), refer tothe paperor the extendedmanual.Getting startedInstallationFor running this code, you will needPython (version >=3.6).
You have two options for installation:If you don't want to modify any code, you can just install the package and use it in commandline:>pipinstallHyBayesAlternatively, if you'd like to make modifications to the code (e.g., the underlying model) you can clone this project. Before running the code, make sure that you have all the requirements by
running the following line to install all the necessary dependencies:>pipinstall-rrequirements.txtIn case you have issue with installing GraphViz, consider using conda installer:>condainstall-canacondagraphviz--yesIf you are using MacOS,brewcommand can used:>brewinstallgraphvizPreparing configuration fileTo analyze your data, you need to prepare the a configuration file that specifies the information needed for the analysis (e.g., the address to your observation files, the type of plots and files you want to be stored).You can find more details on this in theextended manual. Additionally, there are examples of complete configurations inconfigsfolder, after runningmake_configs.py.Running the AnalysisLet's assume that you have followed the above instructions and:(1) you have included the performances observations in two separate files,(2) and you have prepared a config file.To run the analysis, you pass the name of the config file as argument:>python-mHyBayes--configmy_config_file.ini--verboseWhen the flag--verboseis on, the details of the analysis will be printed in standard output.
This flag does not affect the log file that is stored along side other outputs.And here is the general usage template which can be accessed using--helpflag at any time:>python-mHyBayes--help
usage:__main__.py[-h][-cCONFIG][-v][--make_configs][--make_data]RunBayesianStatisticsTailoredtowardsanalysingtheexperimentresults
speciallyinNLParea.Emailesamath@gmail.comforcomments.

optionalarguments:-h,--helpshowthishelpmessageandexit-cCONFIG,--configCONFIGaddressofConfigfile-v,--verboseprintsthereportofthesteps--make_configsifon,exampleconfigurationfileswillbemade.--make_dataifon,exampledatafileswillbemade.ExamplesTo demonstrate everything with a quick example, run the following command to generate toy data:>python-mHyBayes--make_dataYou can see that some artificial data is made in directoryartificial_data. Moreover either use the config files provided in the repository [], or run:>python-mHyBayes--make_configsAnd then execute the analysis on any of the newly made config files. For example:>python-mHyBayes--configconfigs/config_metric.ini--verboseFurther ReadingIf you want to learn more about this package, please refer to theextended manual.
If you want to learn about the concepts discusssed here, refer to the paper below.LicenseThis code is published underApache 2.0license.CitationIf you used this package in your research feel free to cite the following paper:@misc{hypothesisAssessment19,
  title={Not All Claims are Created Equal: Choosing the Right Approach to Assess Your Hypotheses},
  author={{Sadeqi Azer}, Erfan and Khashabi, Daniel and Sabharwal, Ashish and Roth, Dan},
  howpublished={arxiv},
  year={2019}
}