Package: bbrc-validator
bbrc-validatorMain Concepts•Commands•Examples•Install•Contributingbbrc-validatoris a Python-based software package that performs automatic quality
assessment of neuroimaging datasets and their processing derivatives, through
collections of "checkpoints".bbrc-validatoris built on two core concepts:TestsandValidators.ATestchecks a specific trait from a given resource (either an imaging
session or a single scan). It asks a specific question whose answer can be
eitherTrueorFalse(eg."Does this MRI scan have a conversion to NIfTI
available?"). As such,Testsmay be seen asunit tests.  ATestclass is defined by two
attributes (passingandfailing) that refer to two "real-life" cases (one
expected to pass theTestand another expected to fail it). In addition, these
attributes are systematically used by theCItesting.AValidatoris a collection ofTestobjects that may be executed against
anyXNATimaging resource (by referring to their experiment
identifiers). Running aValidatoron a given experiment takes its associated
set of tests, runs them sequentially and collects their results in a JSON object.
A human-readable report can be  generated (as a PDF document) with the results
of the whole procedure.Main ConceptsTest:classMyTest():""" Test functionality description """passing='PASSING_CASE_ID'failing='FAILING_CASE_ID'defrun():# executes the Test logic and returns some ResultsreturnResults(has_passed=test_outcome,data=some_data)defreport():# provides a human-readable version of Results dataValidator:classMyValidator():def__init__():self.tests=[MyTest,...]defrun():# runs all Tests sequentiallydefdump():# compiles all Test results in a single JSON objectdefreport():# generates a human-readable PDF report based on the resultsResult: Represents the outcome from the execution of aTest. It includes
a boolean attributehas_passed(representing the outcome ofTestexecution) and some additionaldataobject (optionally used for
storing contextual information from the execution).Commandsrun_validator.pyExecutes the specifiedValidatoragainst a given image resource (a.k.a XNATexperiment) and generates (a) a JSON object with the results of all the Tests
and (b) a human-readable PDF report.usage: run_validator.py [-h] --config CONFIG --experiment EXPERIMENT
                        [--validator VALIDATOR] --output OUTPUT [--verbose]

Run a validator against an experiment

optional arguments:
  -h, --help                             show this help message and exit
  --config CONFIG, -c CONFIG             XNAT configuration file
  --experiment EXPERIMENT, -e EXPERIMENT XNAT experiment unique identifier
  --validator VALIDATOR, -v VALIDATOR    Validator name (default:ArchivingValidator)
  --output OUTPUT, -o OUTPUT             PDF file to store the report
  --verbose, -V                          Display verbosal information (optional)validation_scores.pyGiven a specific type ofValidator, collects all results available in an XNAT
instance and compiles them in a CSV file.usage: validation_scores.py [-h] --config CONFIG --version VERSION
                            [--validator VALIDATOR] --output OUTPUT 
                            [--project PROJECT] [--verbose]

Compile validation scores

optional arguments:
  -h, --help                    show this help message and exit
  --config CONFIG               XNAT configuration file
  --version VERSION, -v VERSION Filter specific version
  --validator VALIDATOR         Validator name (default:ArchivingValidator)                      
  --output OUTPUT, -o OUTPUT    CSV output file
  --verbose, -V                 Display verbosal information (optional)Enables the creation of tables such as the following example obtained fromArchivingValidator(table trimmed to fit the dimensions of the page).Testsincluded:HasUncompressedPixelDataHasCorrectSequencesHasBvecBvalIsClassicDICOMHasDuplicatedSequencesHasNiftiHasPhilipsPrivateTagsIsStudyDescriptionCorrectTests#1#2#3#4#5#6#7#8Sums1111011116011BBRCDEV_E00211![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)BBRCDEV_E00210![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)BBRCDEV_E00213![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)BBRCDEV_E00212![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)BBRCDEV_E00196![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)BBRCDEV_E00214![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)BBRCDEV_E00217![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)BBRCDEV_E00216![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)BBRCDEV_E00219![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)BBRCDEV_E00218![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)BBRCDEV_E00198![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)![#f03c15](https://via.placeholder.com/15/f03c15/000000?text=+)![#c5f015](https://via.placeholder.com/15/c5f015/000000?text=+)ExamplesCreate a Validator and review the list of its TestsSet apyxnat connectionto the XNAT instance hosting the data requiring validation.Create an instance ofSPM12SegmentValidator, aValidatorfor segmentations
produced usingSPM12 Segment.Print out a list of included tests.importpyxnatintf=pyxnat.Interface(config='.xnat.cfg')frombbrc.validationimportSPM12SegmentValidatorspmv=SPM12SegmentValidator(lut={},xnat_instance=intf)print('{}tests (`{}`):'.format(spmv.__class__.__name__,spmv.version))spmv.testsSPM12SegmentValidator tests (`d6ca22c1`):

[<bbrc.validation.processing.spm.HasCorrectNumberOfItems at 0x273dee24e88>,
 <bbrc.validation.processing.spm.HasCorrectItems at 0x273dee247c8>,
 <bbrc.validation.processing.spm.HasCorrectSPMVersion at 0x273dda8f4c8>,
 <bbrc.validation.processing.spm.HasCorrectMatlabVersion at 0x273dee28848>,
 <bbrc.validation.processing.spm.HasCorrectOSVersion at 0x273dee287c8>,
 <bbrc.validation.processing.spm.SPM12SegmentSnapshot at 0x273dee249c8>,
 <bbrc.validation.processing.spm.HasNormalSPM12Volumes at 0x273dee28788>,
 <bbrc.validation.processing.spm.SPM12SegmentExecutionTime at 0x273dee28bc8>]RunSPM12SegmentValidatoragainst an MRI session,spmv.run('XNAT_E00001')2021-02-04 12:12:54,635 - root - INFO - Running <bbrc.validation.processing.spm.HasCorrectNumberOfItems object at 0x00000273DEE24E88>
2021-02-04 12:12:54,964 - root - ERROR - XNAT_E00001 has 15 items (different from 16)
2021-02-04 12:12:55,572 - root - INFO - Running <bbrc.validation.processing.spm.HasCorrectItems object at 0x00000273DEE247C8>
2021-02-04 12:12:56,120 - root - INFO - Running <bbrc.validation.processing.spm.HasCorrectSPMVersion object at 0x00000273DDA8F4C8>
2021-02-04 12:12:56,592 - root - INFO - Running <bbrc.validation.processing.spm.HasCorrectMatlabVersion object at 0x00000273DEE28848>
2021-02-04 12:12:56,782 - root - INFO - Running <bbrc.validation.processing.spm.HasCorrectOSVersion object at 0x00000273DEE287C8>
2021-02-04 12:12:57,001 - root - INFO - Running <bbrc.validation.processing.spm.SPM12SegmentSnapshot object at 0x00000273DEE249C8>
2021-02-04 12:13:04,997 - root - INFO - * Creating snapshots...
2021-02-04 12:13:46,472 - root - INFO - Saved in /tmp/tmp3j664u27.png
2021-02-04 12:13:46,515 - root - INFO - Running <bbrc.validation.processing.spm.HasNormalSPM12Volumes object at 0x00000273DEE28788>
2021-02-04 12:13:50,552 - root - INFO - Running <bbrc.validation.processing.spm.SPM12SegmentExecutionTime object at 0x00000273DEE28BC8>Collect results fromSPM12SegmentValidatorexecution,importjsonresult=spmv.dump()json.loads(result){'experiment_id': 'XNAT_E00001',
 'version': 'd6ca22c1',
 'generated': '2021-02-04, 12:13',
 'HasCorrectItems': {'has_passed': False,
  'data': ["Missing SPM12_SEGMENT items: ['pyscript_setorigin.m']."]},
 'HasCorrectSPMVersion': {'has_passed': True, 'data': []},
 'HasCorrectMatlabVersion': {'has_passed': True, 'data': []},
 'HasCorrectOSVersion': {'has_passed': True, 'data': []},
 'SPM12SegmentSnapshot': {'has_passed': True, 
  'data': ['/tmp/tmp3j664u27.png']},
 'HasNormalSPM12Volumes': {'has_passed': True,
  'data': ['Volumes: 773592.1702940931 524339.7480925963']},
 'SPM12SegmentExecutionTime': {'has_passed': True, 'data': ['0:07:15']}}Generate a human-readable PDF report from the results,importtempfile_,fp=tempfile.mkstemp(suffix='.pdf')spmv.report(fp)print('Report created:{}'.format(fp))Loading pages (1/6)
Counting pages (2/6)                                               
Resolving links (4/6)                                                       
Loading headers and footers (5/6)                                           
Printing pages (6/6)
Done                                                                      

Report created: '/home/jhuguet/notebooks/bbrc-validator/tmpcexwvwj5.pdf'Installbbrc-validatorcan be installed viapip,pipinstallbbrc-validatorbbrc-validatorrequireswkhtmltopdffor PDF
generation. A static build release (with QT patches) is recommended, see
available releaseshereby OS/distribution.On Ubuntu 18.04:wgethttps://github.com/wkhtmltopdf/packaging/releases/download/0.12.6-1/wkhtmltox_0.12.6-1.bionic_amd64.deb
dpkg-iwkhtmltox_0.12.6-1.bionic_amd64.deb
apt--fix-broken-yinstallOn CentOS 7:wgethttps://github.com/wkhtmltopdf/packaging/releases/download/0.12.6-1/wkhtmltox-0.12.6-1.centos7.x86_64.rpm
yum-ylocalinstallwkhtmltox-0.12.6-1.centos7.x86_64.rpmContributingbbrc-validatoris still under active development. The currently includedTestsandValidatorshave been tailored to the particular needs and context of the
Barcelonaβeta Brain Research Center and as such might differ with the needs from
other organizations.However, the software was designed with an aim towards genericity, modularity and
reusability. Since all Tests are based upon the same template (eg. each of them
being linked to XNAT data resources as test cases), this makes them virtually
shareable across groups and makesbbrc-validatoropen to public contributions.Contact us for details on how to contribute or open an issue to start a discussion.