Package: bandbox
bandboxbandboxis a CLI tool to quickly evaluate how organised your dataset is. You can usebandboxon any directory and it will recursively descend through to all subdirectories to assess whether the structure and names are easy to use.bandboxreads in a set of configs a config file (bandbox.cfg), which it uses to assess your dataset. Once complete,bandboxdisplays various assessments and whether they areokornok, together with details about which paths from the specified directory have failed that assessment. The user may also display the directory tree using the--show-treeoption as well as show all results using the-a/--all; by default, a summary of analysis results is displayed, which modified using the--summarise-sizeoption.:bulb:Tip: Bear in mind thatbandboxonly makes suggestions on how to make your dataset more usable; you are in the best position to decide whethernokis a real problem. It might help you discover unwanted files e.g.logfiles, temporary files such as*.tif~, operating system fluff like.DS_Storefiles on macOS etc.InstallingbandboxInstall from source into a virtualenv using:~$pipinstallgit+https://github.com/emdb-empiar/bandboxor from PyPI using:~$pipinstallbandboxConfiguringbandboxWhen you first try to runbandboxyou will get an error message like this:~$bandbox
error:noconfigsfound;pleasesetBANDBOX_CONFIGenvvarorprovide--config-filepath
info:copyandmodifytheconfigfilefromhttps://raw.githubusercontent.com/emdb-empiar/bandbox/master/bandbox.cfgDownload the config file from the link provided in the error message and save it asbandbox.cfgin your current directory.# if you use wget~$wgethttps://raw.githubusercontent.com/emdb-empiar/bandbox/master/bandbox.cfg# if you use curl~$curlhttps://raw.githubusercontent.com/emdb-empiar/bandbox/master/bandbox.cfg>bandbox.cfg# you can either reference the config file using the --config-file option~$bandbox--config-filebandbox.cfg# or set the BANDBOX_CONFIG environment variable; make sure to use the full path~$exportBANDBOX_CONFIG=`realpathbandbox.cfg`# add the `export BANDBOX_CONFIG=`realpath bandbox.cfg`` line to your ~/.bashrc or ~/.zshrc fileExample usageThis is what an 'ok' dataset would look like:and this is what a 'nok' one would look like:Viewing the treeUse theviewcommand to view the tree implied by the dataset.~$bandboxview# to view the current directory~$bandboxviewsome_path
~$bandboxviewsome_path--prefixsome_path# will exclude 'some_path' from every path entry~$bandboxviewsome_path--hide-file-counts# only show file totals:warning:Warning: Some options are experimental and incomplete e.g.--input-file, which takes the output of Python'sglob.glob(path, recursive=True)function saved as a string.Here is an example based on thetest_datadirectory in the git repository:~$bandboxviewtest_data
└──test_data└──0_bad└──data└──AUThöretal-Averylongrelevanttitlethathasmostofthekeywordsinyourpaper└──AFolderwithanoveralldescription└──9824080293-ControlTissue└──9824080293ControlTissue-Raw└──[2000files:tif=1000;jpg=1000;]└──9824080293ControlTissue-Segmentation└──[7files:zip=7;]└──3789392082-Treatment2Tissue└──3789392082Treatment2Tissue-Raw└──3789392082raw└──[1000files:tif=1000;]└──3789392082Treatment2Tissue-Segmentation└──[7files:zip=7;]└──9028340982-Treatment1Tissue└──9028340982Treatment1Tissue-Raw└──[1000files:tif=1000;]└──9028340982Treatment1Tissue-Segmentation└──[7files:zip=7;]└──0923480928-TreatementTr1-323Tissue└──0923480928TreatementTr1-323Segmentation└──[6files:zip=6;]└──0923480928TreatementTr1-323Raw└──[1000files:tif=1000;]└──folder_with_multiple_folders└──folder5└──[10files:tif=10;]└──folder2└──[10files:tif=10;]└──folder3└──[10files:tif=10;]└──folder4└──[10files:tif=10;]└──folder1└──[10files:tif=10;]└──folder6└──[100files:dog=100;]└──folder7└──[2001files:jpeg=2001;]└──0_good└──[1file:md=1;]└──brief_description└──control_tissue└──segmentation└──[7files:zip=7;]└──raw└──[1000files:tif=1000;]└──treatment3_tissue└──segmentation└──[6files:zip=6;]└──raw└──[1000files:tif=1000;]└──treatment1_tissue└──segmentation└──[7files:zip=7;]└──raw└──[1000files:tif=1000;]└──treatment2_tissue└──segmentation└──[7files:zip=7;]└──raw└──[1000files:tif=1000;]└──folder_with_single_file└──folder└──[1file:txt=1;]└──folder_with_date_name_files└──[12files:txt=10;raw=2;]└──folder_with_multiple_files└──folder└──[12files:txt=10;tif=1;md=1;]└──folder_with_multiple_file_types└──folder└──[20files:txt=10;tif=10;]└──files└──inner_folder└──empty_folder└──folder└──single_empty_folder└──folder└──inner_folder└──folder_with_long_name_folders└──afolderwithspacesinthename└──[1file:mrc=1;]└──folder└──[26files:dog=1;txt=11;tif=10;jpg=2;onx=1;wrx=1;]└──files└──inner_folder└──another_very_long_name_that_we_are_still_wondering_ever_found_the_light_of_day└──afolderwith&funnysymbolsinthe??name└──a_folder_with_a_very_long_name_that_we_cannot_even_begin_to_comprehend└──a.folder.with.periods.in.the.name└──folder_with_non_ascii_characters└──wïth_ñõn_æšçiį└──dataset└──datasets└──with_asciiAnalysing the treeUse theanalysecommand to run the assessments on your dataset. Here is an example output using--show-treeoption:~$bandboxanalysetest_data--show-tree# displays the tree then the summarised resultsModifying configsbandboxuses simple heuristics to analyse a dataset most of which run regular expressions against elements of the tree. A lot of these are raw regex components that are outlined in the config file, which may either be specified using theBANDBOX_CONFIGor--config-fileoption.Once you have a better idea of howbandboxworks, feel free to modify the config file to suit your needs. The config file is divided into two sections:[bandbox]and[regex]. The[bandbox]section contains the configs which are used to assess your dataset. The[regex]section contains the regular expressions which are used to assess your dataset.Here is a summary (annotated) of the current configs:[bandbox]file_extensions=jpg|jpeg|mrc|mrcs|tif|tiff|dm4|txt|box|cfg|fixed|st|rec|map|bak|eer|bz2|gz|zip|xml|am|star|raw|datobvious_files=file|files|data|folder|inner_folder|images|directory# per directorymax_files=2000max_name_length=50# date_infix_chars must start with '-'date_infix_chars=-:/.month_chars=jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec|january|february|march|april|may|june|july|august|september|october|november|decemberaccession_names=EMPIAR|EMDBodd_chars=&?! ,periods_in_name_fewer_than=2external_refs=figure|supplementary[regex]file_re=(?i)^([^.]*\.[^.]*|.*\.(${bandbox:file_extensions}))$$file_extension_re=(?i)^.*\.(${bandbox:file_extensions})$$obvious_files_re=(?i)^(${bandbox:obvious_files})$$accession_names_re=(?i)^.*(${bandbox:accession_names}).*$$odd_chars_re=.*[${bandbox:odd_chars}].*periods_in_name_fewer_than_re=.*([.].*){${bandbox:periods_in_name_fewer_than},}.*external_refs_re=(?i)^.*(${bandbox:external_refs}).*$$# 12/31/2000 or 31/12/2000# 2000[]12[]31 or 2000[]31[]12# 31[]12[]00# 31[]Dec[]2000# Dec[]31[]2000date_re=(?i)^.*\d{2}[${bandbox:date_infix_chars}]*\d{2}[${bandbox:date_infix_chars}]*\d{4}.*$$,(?i)^.*\d{4}[${bandbox:date_infix_chars}]*\d{2}[${bandbox:date_infix_chars}]*\d{2}.*$$,(?i)^.*\d{2}[${bandbox:date_infix_chars}]*\d{2}[${bandbox:date_infix_chars}]*\d{2}.*$$,(?i)^.*\d{2}[${bandbox:date_infix_chars}]*(${bandbox:month_chars})[${bandbox:date_infix_chars}]*\d{4}.*$$,(?i)^.*\d{4}[${bandbox:date_infix_chars}]*(${bandbox:month_chars})[${bandbox:date_infix_chars}]*\d{2}.*$$Interested in contributing?Do you have ideas on other heuristics that can be used to improve the organisation of your data? For example, wouldn't it be cool to infer folder names which are closely related but which have different spellings or typos e.g.tomos,tomogramsandTomogramsin the same dataset probably refer to the same kind of data and could simply all be calledtomograms.Please get in touch then we can provide a step-by-step guide to do so.