Package: hifis-surveyval
HIFIS Surveyval FrameworkWhat Is the Framework Intended For?TheHIFIS Surveyval Frameworkis a tool to analyse data fromLimeSurveydata exports.
It is meant to fill the gap betweenLimeSurveyandPythondata analysis and
visualisation libraries such asNumpy,PandasandMatplotlib.
It is not meant to replace them.
We aim at making life easier for people who are about to conductLimeSurveysurveys and who want to analyse the resulting survey data by providing a tested
and working set of features, so that you need not reinvent the wheel for your
data analysis.
We recommend getting familiar with the framework before the survey is planned
and conducted to make sure that your use case can be covered by the framework.What Do You Need to Provide Yourself?Metadata of Your SurveyThe metadata that is describing your survey such as the list of questions and
answer options including their translations that arenotincluded in theLimeSurveydata export file are given in a metadataYAMLfile that you
need to provide by yourself.
Please have a look into the example metadataYAMLfile given in this project
to get an idea how this file is structured and which information need to be put
there.Pre-processing Steps on Raw DataBefore the analysis is started, the raw data is pre-processed by a
pre-processing script that you need to write by yourself as well
if data cleaning and other pre-processing steps need to be done for your
survey data beforehand.
In order to get an impression how the pre-processing script can look like
please see the example pre-processing scripts provided in this project to get
you started.Analysis Scripts for Your Survey Data AnalysisUnder the hood the architecture of the framework is executing a series of
independent data analysis scripts written by yourself in an analysis run.
As a result, output will be generated in form ofterminal output on the CLI,plots in different formats andtext files.There are also example analysis scripts available in this project that you can
take a look at to understand how these analysis scripts make use of the
framework.Your Use Cases Are Not Covered?As the name of the framework suggests it has been developed to aid in the
analysis of HIFIS surveys conducted for the Helmholtz Federated IT Services
(HIFIS) platform.
Please be aware that as we try to develop a generic framework that can be
used in many use cases, it might still be the case that the implemented
features arose from our use cases and are not applicable in your case.
If this is the case, please get in touch with us, so we can make things happen
for you.
You are also most welcome to contribute to the project with issues, bug fixes,
feature implementations, documentations and testing.
Thank you very much in advance for your valuable contributions!Table of ContentInstallationGetting StartedStart Analysis from Command-Line-InterfaceContribute with Own Analysis ScriptsResourcesAuthor InformationContributorsLicenseGetting StartedThe project's documentation contains a section to help you as auser of the analysis scriptsto run the analysis scripts or as adeveloper of the frameworkto set up the development environment.Getting Started for UsersInstallationTo install the package locally, you can usePip.pipinstallhifis-surveyvalAfter the installation, you can use the tool from the command line withhifis-surveyval --help.Getting Started for DevelopersInstallationTo install the package locally, you can usePoetry.Using PoetryIf you want to actively contribute changes to the project, you are required to
also install the development packages alongside the framework.gitclonehttps://gitlab.hzdr.de/hifis/overall/surveys/hifis-surveyval.gitcdhifis-surveyval
poetryinstallAfter the installation, you can use the tool from the command line withpoetry run hifis-surveyval --helpPoetry installs some packages that are required for performing quality checks.
Usually they are also performed via GitLab CI, but can also be executed locally.It is common practice to run some checks locally before pushing them online.
Therefore, execute below commands:$# Order your imports$isort-rc.$makelintThe following documentation references the pip installation.
You can use the same commands with a poetry installation, if you prefix your
commands withpoetry run COMMAND.Start Analysis from Command-Line-InterfaceThe survey analysis package is a program to be executed on the
Command-Line-Interface (CLI).Quick Start Example: Run AnalysisDue to sensible defaults of the project's configurations you need to have the
analysis scripts, the preprocessing script as well as metadata and data files
in certain locations in order to run the survey analysis.
This configuration filehifis-surveyval.ymlwhich includes these defaults is
created with the commandhifis-surveyval init.
Please put your analysis scripts into a sub-folder calledscripts.
The preprocessing scriptpreprocess.pyis expected in the root folder of the
project.
Make sure that the filemeta.ymlis put into sub-foldermetadata.
Finally, copy the CSV data file of your survey to a central location like adatasub-folder and tell the program the path to that data file on the
command line when running the survey analysis.Now you can do the following to start the survey analysis from the CLI:hifis-surveyvalanalyzedata/<data_file_name>.csvThe output is then put into a sub-folder within the folderoutputwhich is named after the stamp of the current date-time if not specified
differently.Caution:Depending on the Operating System used an issue with the file
encoding might occur.
There might be data-CSV-files around which are encoded withUTF-8-BOMwhich causes errors when read in on Windows OS.
In this case you need to change the encoding toUTF-8before running
the survey analysis.FlagsThe program accepts two flags:Help flagVerbosity flagHelp flagCalling the program with thehelp-flag is the first thing to do
when being encountered with this program.
It outputs a so-calledUsage-message to the CLI:$hifis-surveyval--helpPlease issue this command on the CLI and read the detailedUsage-message before continuing with reading the documentation
of theUsage-message here.Verbosity flagTheverbosity-flag can be provided in order to specify the verbosity
of the output to the CLI.
This flag is called--verboseor-vfor short:hifis-surveyval--verbose<COMMAND>hifis-surveyval-v<COMMAND>The verbosity of the output can be increased even more
by duplicating the flag--verboseor-vup to two times:hifis-surveyval--verbose--verbose--verbose<COMMAND>hifis-surveyval-vvv<COMMAND>CommandsThere are three different commands implemented which come with its own set of
flags and parameters:CommandversionCommandinitCommandanalyzeCommandversionTheversioncommand outputs the version number of this CLI-program like so:hifis-surveyvalversionCommandinitBefore you start the analysis you may want to change the defaults of the
configuration variables.
In order to do so, you can create a configuration file that is namedhifis-surveyval.ymlby issuing theinitcommand:hifis-surveyvalinitThis file contains the following information:ANONYMOUS_QUESTION_ID:_DATA_ID_SEPARATOR:_HIERARCHY_SEPARATOR:/ID_COLUMN_NAME:idMETADATA:metadataOUTPUT_FOLDER:outputOUTPUT_FORMAT:SCREENPREPROCESSING_FILENAME:preprocess.pySCRIPT_FOLDER:scriptsSCRIPT_NAMES:[]CUSTOM_PLOT_STYLE:"report_style"# OptionalConcepts that You Need to KnowQuestionCollection: This concept refers to a set ofQuestionsthat
cover the same topic.Question: This concept refers to an atomicQuestionthat can not be on
its own and needs to be wrapped up into aQuestionCollection.Note:Other terms that may describe similar concepts arequestion(which equals toQuestionCollection) andsub-question(which equals toQuestion).Configuration File Entries ExplainedANONYMOUS_QUESTION_IDdefines a placeholder for Question IDs. The CSV data
might not explicitly mention a full ID of a Question but solely the
QuestionCollection ID. In this case the HIFIS Surveyval Framework adds a
character, by default_(underscore), to the QuestionCollection ID to mark
this situation.The CSV data file is structured into header and body rows. The header row
consists of a comma-separated list of column names. Some column names contain
a separator character that concatenates the QuestionCollection ID with the
Question ID, theDATA_ID_SEPARATOR. This variable indicates which character
is used to separate these IDs. If not specified otherwise, it defaults to_(underscore).ThisDATA_ID_SEPARATORcharacter is internally replaced by a different
character, the so calledHIERARCHY_SEPARATOR, which defaults to a/(slash).WithID_COLUMN_NAMEyou may want to specify the name of theidcolumn in
the CSV data file.Each analysis needs metadata about the questions asked in the survey and
answers that participants may give. SettingMETADATAspecifies the location
of the metadata files which are by default located in a folder calledmetadata. Be aware that it is recommended to have one YAML file per
QuestionCollection. Each YAML file then covers the metadata of a single
QuestionCollection and is named according to the ID of this
QuestionCollection.You may specify the output folder by settingOUTPUT_FOLDERwhich is namedoutputby default.You may prefer a specific output format likePDF,PNG,SVGorSCREENwhich you may select viaOUTPUT_FORMAT.
The default value isSCREEN.
Note: Be aware that other output formats like text or markdown files may be
created, which depends largely upon the implementation of the analysis
scripts.You might want to tell the program where to find the preprocessing filepreprocess.pythat preprocesses and filters your survey data according to
specific rules.
You can do so by settingPREPROCESSING_FILENAME.You may specify the folder which contains the analysis scripts with settingSCRIPT_FOLDER, which is thescriptsfolder by default.WithSCRIPT_NAMESyou may select a subset of the analysis scripts available
as a list that ought to be executed.
This list is empty by default, which means, all scripts are executed.Hint for LimeSurvey UsersThere is an optionExpression Manager codein LimeSurvey when exporting the
data into a CSV file that uses the separator character_(underscore) to
concatenate QuestionCollection ID and Question ID in the CSV data header.
Otherwise the default[]is used which is not compatible with the HIFIS
Surveyval Framework.Additional Files GeneratedAdditional to the configuration file, three more files are created:Filepreprocess.pyis created in the root folder of the project.Fileexample_script.pyis created in thescriptsfolder of the project.Filestyle_template.mplstyleis created in thecustom_plot_stylesfolder of the project.CommandanalyzeThe more interesting command is theanalyzecommand
which comes with adata-parameter.
Thedata-parameter cannotbe omitted and need to be given explicitly
in order to be able to start the analysis.
This is an example of how to do the analysis:hifis-surveyvalanalyzedata/<data_file_name>.csvContribute with Own Analysis ScriptsEssential Requirements for Developing Own Analysis ScriptsAs you might have read in the previous sections the actual analysis scripts
reside in a specific folder calledscripts.
All scripts in that folder will be automatically discovered by the packagehifis-surveyvalwhen running the analysis.
In order that the program recognizes the scripts in that folder as
analysis scripts they need to fulfill the following two criteria:The filename need to end on.py.The file need to contain a function calledrunwithout any parameters."""A dummy script for testing the function dispatch.. currentmodule:: hifis_surveyval.scripts.dummy.. moduleauthor:: HIFIS Software <software@hifis.net>"""defrun():print("Example Script")If both requirements are satisfied the program will execute therun-functions
of the analysis scripts in an arbitrary order.File-System Structure of the Core Component$treehifis_surveyval/
hifis_surveyval
├──cli.py
├──core
│├──dispatch.py
│├──preprocess.py
│├──settings.py
│└──util.py
├──data_container.py
├──files
│├──example_preprocess.py
│├──example_script.py
│├──preprocess.py
│├──scripts
││└──example-01-accessing-data.py
│└──style_template.mplstyle
├──hifis_surveyval.py
├──models
│├──answer_option.py
│├──answer_types.py
│├──mixins
││├──mixins.py
││├──uses_settings.py
││└──yaml_constructable.py
│├──question_collection.py
│├──question.py
│└──translated.py
├──plotting
│├──matplotlib_plotter.py
│├──plot_styles
││└──report_style.mplstyle
│├──plotter.py
│└──supported_output_format.py
└──printing└──printer.pyMatplotlib Custom Plot StylesFiles with file ending.mplstylecontain styling information forMatplotlibplots.
An example Matplotlib plot style template calledstyle_template.mplstyleis
created with theinit-command of theHIFIS-Surveyvalframework.
These files can be put in a folder calledcustom_plot_styles.
Custom plot style files that are generally used within the framework need to be
put into folderhifis_surveyval/plotting/plot_styles/.
Plots created withMatplotlibaccept akwargs-argumentplot_style_namewhich chooses which plot style to use for the plot.
Styles in foldercustom_plot_styleshave precedence over styles in folderhifis_surveyval/plotting/plot_styles/if they have the same name.ResourcesBelow are some handy resource links:Project DocumentationClickis a Python package for creating beautiful command line interfaces in a composable way with as little code as necessary.Sphinxis a tool that makes it easy to create intelligent and beautiful documentation, written by Geog Brandl and licnsed under the BSD license.pytesthelps you write better programs.GNU Makeis a tool which controls the generation of executables and other non-source files of a program from the program's source files.Author InformationHIFIS-Surveyvalwas created byHIFIS Software Services.ContributorsWe would like to thank and give credits to the following contributors of this
project:Be the first to be named here!LicenseCopyright © 2021 HIFIS Softwaresupport@hifis.netThis work is licensed under the following license(s):Everything else is licensed underGPL-3.0-or-laterPlease see the individual files for more accurate information.Hint:We provided the copyright and license information in accordance to theREUSE Specification 3.0.