Package: pynidm
1PyNIDM: Neuroimaging Data Model in PythonA Python library to manipulate theNeuroimaging Data Model.Contents1PyNIDM: Neuroimaging Data Model in Python1.1Dependencies1.2Installation1.3Contributing to the Software1.4Reporting Issues or Problems1.5Support and Feedback1.6NIDM-Experiment Tools1.6.1BIDS MRI Conversion to NIDM1.6.2CSV File to NIDM Conversion1.6.3convert1.6.4concatenate1.6.5visualize1.6.6merge1.6.7Query1.6.8linear_regression2PyNIDM: REST API and Command Line Usage2.1Introduction2.2Installation2.3URI formats2.3.1Operations2.3.2Query Parameters2.4Return Formatting2.4.1Examples2.4.1.1Get the UUID for all the projects at this location2.4.1.2Get the project summary details2.4.1.3Get the subjects in a project2.4.1.4Use the command line to get statistics on a project for the AGE_AT_SCAN and a FSL data element2.4.1.5Get details on a subject2.5Additional NIDM-related Tools1.1DependenciesGit-annexGraphviz(native package):Fedora:dnf install graphvizOS-X:brew install graphviz1.2Installation$pipinstallpynidm1.3Contributing to the SoftwareThis software is open source and community developed.  As such, we encourage
anyone and everyone interested in semantic web and neuroimaging to contribute.
To begin contributing code to the repository, please fork the main repo into
your user space and use the pull request GitHub feature to submit code for
review.  Please provide a reasonably detailed description of what was changed
and why in the pull request.1.4Reporting Issues or ProblemsIf you encounter a bug, you can directly report it in the issues section.
Please describe how to reproduce the issue and include as much information as
possible that can be helpful for fixing it. If you would like to suggest a fix,
please open a new pull request or include your suggested fix in the issue.1.5Support and FeedbackWe would love to hear your thoughts on our Python toolbox. Feedback, questions,
or feature requests can also be submitted as issues. Note, we are a small band
of researchers who mostly volunteer our time to this project.  We will respond
as quickly as possible.1.6NIDM-Experiment Tools1.6.1BIDS MRI Conversion to NIDMThis program will convert a BIDS MRI dataset to a NIDM-Experiment RDF document.
It will parse phenotype information and simply store variables/values and link
to the associated json data dictionary file.  To use this tool please set your
INTERLEX_API_KEY environment variable to your unique API key.  To get an
Interlex API key you visitSciCrunch,
register for an account, then click on “MyAccount” and “API Keys” to add a new
API key for your account.$bidsmri2nidm-d[ROOTBIDSDIRECT]-bidsignoreusage:bidsmri2nidm[-h]-dDIRECTORY[-jsonld][-bidsignore][-no_concepts][-json_mapJSON_MAP][-logLOGFILE][-oOUTPUTFILE]ThisprogramwillrepresentaBIDSMRIdatasetasaNIDMRDFdocumentandprovideuserwithopportunitytoannotatethedataset(i.e.createsidecarfiles)andassociateselectedvariableswithbroaderconceptstomakedatasetsmoreFAIR.Note,youmustobtainanAPIkeytoInterlexbysigningupforanaccountatscicrunch.orgthengoingtoMyAccountandAPIKeys.ThensettheenvironmentvariableINTERLEX_API_KEYwithyourkey.optionalarguments:-h,--helpshowthishelpmessageandexit-dDIRECTORYFullpathtoBIDSdatasetdirectory-jsonld,--jsonldIfflagset,outputisjson-ldnotTURTLE-bidsignore,--bidsignoreIfflagset,toolwilladdNIDM-relatedfilesto.bidsignorefile-no_concepts,--no_conceptsIfflagset,toolwillnodoconceptmapping-logLOGFILE,--logLOGFILEFullpathtodirectorytosavelogfile.Logfilenameisbidsmri2nidm_[basename(args.directory)].log-oOUTPUTFILEOutputsturtlefilecallednidm.ttlinBIDSdirectorybydefault..orwhateverpath/filenameissetheremapvariablestotermsarguments:-json_mapJSON_MAP,--json_mapJSON_MAPOptionalfullpathtouser-suppledJSONfilecontainingdataelementdefintitions.1.6.2CSV File to NIDM ConversionThis program will load in a CSV file and iterate over the header variable names
performing an elastic search ofhttps://scicrunch.org/nidm-termsfor
NIDM-ReproNim tagged terms that fuzzy match the variable names. The user will
then interactively pick a term to associate with the variable name. The
resulting annotated CSV data will then be written to a NIDM data file.  To use
this tool please set your INTERLEX_API_KEY environment variable to your unique
API key.  To get an Interlex API key you visitSciCrunch, register for an account, then click on
“MyAccount” and “API Keys” to add a new API key for your account.usage:csv2nidm[-h]-csvCSV_FILE[-json_mapJSON_MAP|-redcapREDCAP][-nidmNIDM_FILE][-no_concepts][-logLOGFILE]-outOUTPUT_FILEThisprogramwillloadinaCSVfileanditerateovertheheadervariablenamesperforminganelasticsearchofhttps://scicrunch.org/forNIDM-ReproNimtaggedtermsthatfuzzymatchthevariablenames.Theuserwilltheninteractivelypickatermtoassociatewiththevariablename.TheresultingannotatedCSVdatawillthenbewrittentoaNIDMdatafile.Note,youmustobtainanAPIkeytoInterlexbysigningupforanaccountatscicrunch.orgthengoingtoMyAccountandAPIKeys.ThensettheenvironmentvariableINTERLEX_API_KEYwithyourkey.ThetoolsupportsimportofRedCapdatadictionariesandwillconvertrelevantinformationintoajson-formattedannotationfileusedtoannotatethedataelementsintheresultingNIDMfile.optionalarguments:-h,--helpshowthishelpmessageandexit-csvCSV_FILEFullpathtoCSVfiletoconvert-json_mapJSON_MAPFullpathtouser-suppledJSONfilecontainingvariable-termmappings.-redcapREDCAPFullpathtoauser-suppliedRedCapformatteddatadictionaryforcsvfile.-nidmNIDM_FILEOptionalfullpathofNIDMfiletoaddCSV->NIDMconvertedgraphto-no_conceptsIfthisflagissetthennoconceptassociationswillbeaskedoftheuser.Thisisusefulifyoualreadyhavea-json_mapspecifiedwithoutconceptsandwanttosimplyrunthisprogramtogetaNIDMfilewithuserinteractiontoassociateconcepts.-logLOGFILE,--logLOGFILEfullpathtodirectorytosavelogfile.Logfilenameiscsv2nidm_[arg.csv_file].log-outOUTPUT_FILEFullpathwithfilenametosaveNIDMfile1.6.3convertThis function will convert NIDM files to various RDF-supported formats and name
then / put them in the same place as the input file.Usage:pynidmconvert[OPTIONS]Options:-nl,--nidm_file_listTEXTAcommaseparatedlistofNIDMfileswithfullpath[required]-t,--type[turtle|jsonld|xml-rdf|n3|trig]IfparametersetthenNIDMfilewillbeexportedasJSONLD[required]--helpShowthismessageandexit.1.6.4concatenateThis function will concatenate NIDM files.  Warning, no merging will be done so
you may end up with multiple prov:agents with the same subject id if you’re
concatenating NIDM files from multiple visits of the same study.  If you want
to merge NIDM files on subject ID see pynidm mergeUsage:pynidmconcat[OPTIONS]Options:-nl,--nidm_file_listTEXTAcommaseparatedlistofNIDMfileswithfullpath[required]-o,--out_fileTEXTFiletowriteconcatenatedNIDMfiles[required]--helpShowthismessageandexit.1.6.5visualizeThis command will produce a visualization(pdf) of the supplied NIDM files named
the same as the input files and stored in the same directories.Usage:pynidmvisualize[OPTIONS]Options:-nl,--nidm_file_listTEXTAcommaseparatedlistofNIDMfileswithfullpath[required]--helpShowthismessageandexit.1.6.6mergeThis function will merge NIDM files.  See command line parameters for supported
merge operations.Usage:pynidmmerge[OPTIONS]Options:-nl,--nidm_file_listTEXTAcommaseparatedlistofNIDMfileswithfullpath[required]-s,--sIfparametersetthenfileswillbemergedbyndar:src_subjec_idofprov:agents-o,--out_fileTEXTFiletowriteconcatenatedNIDMfiles[required]--helpShowthismessageandexit.1.6.7QueryThis function provides query support for NIDM graphs.Usage:pynidmquery[OPTIONS]Options:-nl,--nidm_file_listTEXTAcommaseparatedlistofNIDMfileswithfullpath[required]-nc,--cde_file_listTEXTAcommaseparatedlistofNIDMCDEfileswithfullpath.CanalsobesetintheCDE_DIRenvironmentvariable-q,--query_fileFILENAMETextfilecontainingaSPARQLquerytoexecute-p,--get_participantsParameter,ifset,querywillreturnparticipantIDsandprov:agententityIDs-i,--get_instrumentsParameter,ifset,querywillreturnlistofonli:assessment-instrument:-iv,--get_instrument_varsParameter,ifset,querywillreturnlistofonli:assessment-instrument:variables-de,--get_dataelementsParameter,ifset,willreturnallDataElementsinNIDMfile-debv,--get_dataelements_brainvolsParameter,ifset,willreturnallbrainvolumeDataElementsinNIDMfilealongwithdetails-bv,--get_brainvolsParameter,ifset,willreturnallbrainvolumedataelementsandvaluesalongwithparticipantIDsinNIDMfile-o,--output_fileTEXTOptionaloutputfile(CSV)tostoreresultsofquery-u,--uriTEXTARESTAPIURIquery-j/-no_jReturnresultofauriqueryasJSON-v,--verbosityTEXTVerbositylevel0-5,0isdefault--helpShowthismessageandexit.Details on the REST API URI format and usage can be found on the REST API usage
page.1.6.8linear_regressionThis function provides linear regression support for NIDM graphs.Usage:pynidmlinear-regression[OPTIONS]Options:-nl,--nidm_file_listTEXTAcomma-separatedlistofNIDMfileswithfullpath[required]-r,--regularizationTEXTParameter,ifset,willreturntheresultsofthelinearregressionwithL1orL2regularizationdependingonthetypespecified,andtheweightwiththemaximumlikelihoodsolution.Thiswillpreventoverfitting.(Ex:-rL1)-model,--mlTEXTAnequationrepresentingthelinearregression.Thedependentvariablecomesfirst,followedby"="or"~",followedbytheindependentvariablesseparatedby"+"(Ex:-model"fs_003343 = age*sex + sex +
                                  age + group + age*group + bmi")[required]-contstant,--ctrTEXTParameter,ifset,willreturndifferencesinvariablerelationshipsbygroup.Oneormultipleparameterscanbeused(multipleparametersshouldbeseparatedbyacomma-separatedlist)(Ex:-contrastgroup,age)-o,--output_fileTEXTOptionaloutputfile(TXT)tostoreresultsofquery--helpShowthismessageandexit.To use the linear regression algorithm successfully, structure, syntax, and
querying is important. Here is how to maximize the usefulness of the tool:First, use pynidm query to discover the variables to use. PyNIDM allows for the
use of either data elements (PIQ_tca9ck), specific URLs
(http://uri.interlex.org/ilx_0100400), or source variables (DX_GROUP).An example of a potential query is:pynidm query -nl /simple2_NIDM_examples/datasets.datalad.org/abide/RawDataBIDS/CMU_a/nidm.ttl,/simple2_NIDM_examples/datasets.datalad.org/abide/RawDataBIDS/CMU_b/nidm.ttl -u /projects?fields=fs_000008,DX_GROUP,PIQ_tca9ck,http://uri.interlex.org/ilx_0100400You can also do:pynidm query -nl /simple2_NIDM_examples/datasets.datalad.org/abide/RawDataBIDS/CMU_a/nidm.ttl,/Users/Ashu/Downloads/simple2_NIDM_examples/datasets.datalad.org/abide/RawDataBIDS/CMU_b/nidm.ttl -gf fs_000008,DX_GROUP,PIQ_tca9ck,http://uri.interlex.org/ilx_0100400The query looks in the two files specified in the -nl parameter for the
variables specified. In this case, we use fs_000008 and DX_GROUP (source
variables), a URL (http://uri.interlex.org/ilx_0100400), and a data element
(PIQ_tca9ck). The output of the file is slightly different depending on whether
you use -gf or -u. With -gf, it will return the variables from both files
separately, while -u combines them.Now that we have selected the variables, we can perform a linear regression. In
this example, we will look at the effect of DX_GROUP, age at scan, and PIQ on
supratentorial brain volume.The command to use for this particular data is:pynidm linear-regression -nl /simple2_NIDM_examples/datasets.datalad.org/abide/RawDataBIDS/CMU_a/nidm.ttl,/simple2_NIDM_examples/datasets.datalad.org/abide/RawDataBIDS/CMU_b/nidm.ttl -model "fs_000008 = DX_GROUP + PIQ_tca9ck + http://uri.interlex.org/ilx_0100400" -contrast "DX_GROUP" -r L1-nl specifies the file(s) to pull data from, while -model is the model to
perform a linear regression model on. In this case, the variables are fs_000008
(the dependent variable, supratentorial brain volume), DX_GROUP (diagnostic
group), PIQ_tca9ck (PIQ), andhttp://uri.interlex.org/ilx_0100400(age at
scan). The -contrast parameter says to contrast the data using DX_GROUP, and
then do a L1 regularization to prevent overfitting.Details on the REST API URI format and usage can be found below.2PyNIDM: REST API and Command Line Usage2.1IntroductionThere are two main ways to interact with NIDM data using the PyNIDM REST API.
First, the pynidm query command line utility will accept queries formatted as
REST API URIs. Second, the rest-server.py script can be used to run a HTTP
server to accept and process requests. This script can either be run directly
or using a docker container defined in the docker directory of the project.Example usage:$pynidmquery-nl"cmu_a.ttl,cmu_b.ttl"-u/projectsdc1bf9be-10a3-11ea-8779-003ee1ce9545ebe112da-10a3-11ea-af83-003ee1ce95452.2InstallationTo use the REST API query syntax on the command line, follow the PyNIDMinstallation instructions.The simplest way to deploy a HTTP REST API server would be with the provided
docker container. You can find instructions for that process in theREADME.mdfile in the docker
directory of the Github repository.2.3URI formatsYou can find details on the REST API at theSwaggerHub API Documentation.  The OpenAPI
specification file is part of the Github repository in
‘docs/REST_API_definition.openapi.yaml’Here is a list of the current operations. See the SwaggerHub page for more
details and return formats.- /projects
- /projects/{project_id}
- /projects/{project_id}/subjects
- /projects/{project_id}/subjects?filter=[filter expression]
- /projects/{project_id}/subjects/{subject_id}
- /projects/{project_id}/subjects/{subject_id}/instruments/{instrument_id}
- /projects/{project_id}/subjects/{subject_id}/derivatives/{derivative_id}
- /statistics/projects/{project_id}You can append the following query parameters to many of the operations:- filter
- field2.3.1Operations/projectsGet a list of all project IDs available.Supported query parameters: none/projects/{project_id}See some details for a project. This will include the list of subject IDs
and data elements used in the projectSupported query parameters: filter/projects/{project_id}/subjectsGet the list of subjects in a projectSupported query parameters: filter/projects/{project_id}/subjects/{subject_id}Get the details for a particular subject. This will include the results of
any instrumnts or derivatives associated with the subject, as well as a
list of the related activities.Supported query parameters: none/projects/{project_id}/subjects/{subject_id}/instruments/{instrument_id}Get the values for a particular instrumentSupported query parameters: none/projects/{project_id}/subjects/{subject_id}/derivatives/{derivative_id}Get the values for a particular derivativeSupported query parameters: none/statistics/projects/{project_id}See project statistics. You can also use this operation to get statsitcs on
a particular instrument or derivative entry by use afieldquery option.Supported query parameters: filter, field/statistics/projects/{project_id}/subjects/{subject_id}See some details for a project. This will include the list of subject IDs
and data elements used in the projectSupported query parameters: none2.3.2Query ParametersfilterThe filter query parameter is used when you want to receive data only on
subjects that match some criteria.  The format for the filter value should
be of the form:identifier op value [ and identifier op value and ... ]Identifiers should be formatted as “instrument.ID” or “derivatives.ID”  You
can use any value for the instrument ID that is shown for an instrument or
in the data_elements section of the project details. For the derivative ID,
you can use the last component of a derivative field URI (ex. for the URIhttp://purl.org/nidash/fsl#fsl_000007, the ID would be “fsl_000007”) or the
exact label shown when viewing derivative data (ex. “Left-Caudate (mm^3)”).Theopcan be one of “eq”, “gt”, “lt”.Example filters:?filter=instruments.AGE_AT_SCANgt 30?filter=instrument.AGE_AT_SCANeq 21 and derivative.fsl_000007 lt 3500fieldsThe fields query parameter is used to specify what fields should be
detailed in a statistics operation. For each field specified the result
will show minimum, maximum, average, median, and standard deviation for the
values of that field across all subjects matching the operation and filter.
Multiple fields can be specified by separating each field with a comma.Fields should be formatted in the same way as identifiers are specified in
the filter parameter.Example field query:http://localhost:5000/statistics/projects/abc123?field=instruments.AGE_AT_SCAN,derivatives.fsl_0000202.4Return FormattingBy default the HTTP REST API server will return JSON formatted objects or
arrays.  When using the pynidm query command line utility the default return
format is text (when possible) or you can use the -j option to have the output
formatted as JSON.2.4.1Examples2.4.1.1Get the UUID for all the projects at this locationcurlhttp://localhost:5000/projectsExample response:["dc1bf9be-10a3-11ea-8779-003ee1ce9545"]2.4.1.2Get the project summary detailscurlhttp://localhost:5000/projects/dc1bf9be-10a3-11ea-8779-003ee1ce9545Example response:{"http://www.w3.org/1999/02/22-rdf-syntax-ns#type":"http://purl.org/nidash/nidm#Project","dctypes:title":"ABIDE CMU_a Site","http://www.w3.org/ns/prov#Location":"/datasets.datalad.org/abide/RawDataBIDS/CMU_a","sio:Identifier":"1.0.1","nidm:NIDM_0000171":14,"age_max":33.0,"age_min":21.0,"ndar:gender":["1","2"],"obo:handedness":["R","L","Ambi"]}2.4.1.3Get the subjects in a projectpynidmquery-nl"cmu_a.nidm.ttl"-uhttp://localhost:5000/projects/dc1bf9be-10a3-11ea-8779-003ee1ce9545/subjectsExample response:deef8eb2-10a3-11ea-8779-003ee1ce9545
df533e6c-10a3-11ea-8779-003ee1ce9545
ddbfb454-10a3-11ea-8779-003ee1ce9545
df21cada-10a3-11ea-8779-003ee1ce9545
dcfa35b2-10a3-11ea-8779-003ee1ce9545
de89ce4c-10a3-11ea-8779-003ee1ce9545
dd2ce75a-10a3-11ea-8779-003ee1ce9545
ddf21020-10a3-11ea-8779-003ee1ce9545
debc0f74-10a3-11ea-8779-003ee1ce9545
de245134-10a3-11ea-8779-003ee1ce9545
dd5f2f30-10a3-11ea-8779-003ee1ce9545
dd8d4faa-10a3-11ea-8779-003ee1ce9545
df87cbaa-10a3-11ea-8779-003ee1ce9545
de55285e-10a3-11ea-8779-003ee1ce95452.4.1.4Use the command line to get statistics on a project for the AGE_AT_SCAN and a FSL data elementpynidmquery-nlttl/cmu_a.nidm.ttl-u/statistics/projects/dc1bf9be-10a3-11ea-8779-003ee1ce9545?fields=instruments.AGE_AT_SCAN,derivatives.fsl_000001Example response:-------------------------------------------------  ---------------------------------------------
"http://www.w3.org/1999/02/22-rdf-syntax-ns#type"  http://www.w3.org/ns/prov#Activity
"title"                                            ABIDE CMU_a Site
"Identifier"                                       1.0.1
"prov:Location"                                    /datasets.datalad.org/abide/RawDataBIDS/CMU_a
"NIDM_0000171"                                     14
"age_max"                                          33.0
"age_min"                                          21.0

  gender
--------
       1
       2

handedness
------------
R
L
Ambi

subjects
------------------------------------
de89ce4c-10a3-11ea-8779-003ee1ce9545
deef8eb2-10a3-11ea-8779-003ee1ce9545
dd8d4faa-10a3-11ea-8779-003ee1ce9545
ddbfb454-10a3-11ea-8779-003ee1ce9545
de245134-10a3-11ea-8779-003ee1ce9545
debc0f74-10a3-11ea-8779-003ee1ce9545
dd5f2f30-10a3-11ea-8779-003ee1ce9545
ddf21020-10a3-11ea-8779-003ee1ce9545
dcfa35b2-10a3-11ea-8779-003ee1ce9545
df21cada-10a3-11ea-8779-003ee1ce9545
df533e6c-10a3-11ea-8779-003ee1ce9545
de55285e-10a3-11ea-8779-003ee1ce9545
df87cbaa-10a3-11ea-8779-003ee1ce9545
dd2ce75a-10a3-11ea-8779-003ee1ce9545

-----------  ------------------  --------
AGE_AT_SCAN  max                 33
AGE_AT_SCAN  min                 21
AGE_AT_SCAN  median              26
AGE_AT_SCAN  mean                26.2857
AGE_AT_SCAN  standard_deviation   4.14778
-----------  ------------------  --------

----------  ------------------  -----------
fsl_000001  max                 1.14899e+07
fsl_000001  min                 5.5193e+06
fsl_000001  median              7.66115e+06
fsl_000001  mean                8.97177e+06
fsl_000001  standard_deviation  2.22465e+06
----------  ------------------  -----------2.4.1.5Get details on a subjectUse-jfor a JSON-formatted responsepynidmquery-j-nl"cmu_a.nidm.ttl"-uhttp://localhost:5000/projects/dc1bf9be-10a3-11ea-8779-003ee1ce9545/subjects/df21cada-10a3-11ea-8779-003ee1ce9545Example response:{"uuid":"df21cada-10a3-11ea-8779-003ee1ce9545","id":"0050665","activity":["e28dc764-10a3-11ea-a7d3-003ee1ce9545","df28e95a-10a3-11ea-8779-003ee1ce9545","df21c76a-10a3-11ea-8779-003ee1ce9545"],"instruments":{"e28dd218-10a3-11ea-a7d3-003ee1ce9545":{"SRS_VERSION":"nan","ADOS_MODULE":"nan","WISC_IV_VCI":"nan","WISC_IV_PSI":"nan","ADOS_GOTHAM_SOCAFFECT":"nan","VINELAND_PLAY_V_SCALED":"nan","null":"http://www.w3.org/ns/prov#Entity","VINELAND_EXPRESSIVE_V_SCALED":"nan","SCQ_TOTAL":"nan","SRS_MOTIVATION":"nan","PIQ":"104.0","FIQ":"109.0","WISC_IV_PRI":"nan","FILE_ID":"CMU_a_0050665","VIQ":"111.0","WISC_IV_VOCAB_SCALED":"nan","VINELAND_DAILYLVNG_STANDARD":"nan","WISC_IV_SIM_SCALED":"nan","WISC_IV_DIGIT_SPAN_SCALED":"nan","AGE_AT_SCAN":"33.0"}},"derivatives":{"b9fe0398-16cc-11ea-8729-003ee1ce9545":{"URI":"http://iri.nidash.org/b9fe0398-16cc-11ea-8729-003ee1ce9545","values":{"http://purl.org/nidash/fsl#fsl_000005":{"datumType":"ilx_0102597","label":"Left-Amygdala (voxels)","value":"1573","units":"voxel"},"http://purl.org/nidash/fsl#fsl_000004":{"datumType":"ilx_0738276","label":"Left-Accumbens-area (mm^3)","value":"466.0","units":"mm^3"},"http://purl.org/nidash/fsl#fsl_000003":{"datumType":"ilx_0102597","label":"Left-Accumbens-area (voxels)","value":"466","units":"voxel"}},"StatCollectionType":"FSLStatsCollection"}}2.5Additional NIDM-related ToolsNIDM-Terms <https://github.com/NIDM-Terms/terms>NIDM-Terms Scicrunch Interface <https://scicrunch.org/nidm-terms>Freesurfer stats -> NIDM <https://github.com/repronim/segstats_jsonld>FSL structural segmentation -> NIDM <https://github.com/ReproNim/fsl_seg_to_nidm>ANTS structural segmentation -> NIDM <https://github.com/ReproNim/ants_seg_to_nidm>