Package: rsbids
rsbidsrsbidsis a rust implementation ofpybids, currently under active development. It offers vastly improved runtimes compared to other bids indexers (benchmarks to come), a streamlined core api, and a pybids compatibility api.rsbidsis currently in alpha. Most of the core pybids features are implemented, however, there is little to no automated testing or documentation. It has only rudimentary validation and no configurability. Pybids compatibility has been implemented for much ofpybids.layout.layout,pybids.layout.indexers, andpybids.layout.models. Not all features are available, however. Whenever possible, aCompatibilityErroror warning will be raised when these features are encountered. Finally, api stability is not guarenteed for any aspect of the api.The alpha period is an opportunity to test and experiment. Community engagement and feedback is highly valued, and will have an impact on future development. In the immediate future, work will focus on testing, stability, and basic configuration/validation. However, any feature ideas and feedback on the api are welcome. (Note that there's a number of issues I'm already aware of, so be sure to read this document before leaving bug reports).Installationrsbidsis precompiled for most environments, so installation is generally as simple as:pipinstallrsbidsOn more exotic linux versions, or custom environments such as HPCs, the precompiled wheels may not work andrsbidswill need to be compiled. Fortunately, this is generally really straight forward.First, ensure rust is installed on your system. You can follow the simple instructions fromrustupto install directly, or on an HPC, load up rust using its software version control (e.g. forlmod:module load rust). Then just pip-install as normal, andrsbidsshould automatically be compiled (note that it may take several minutes).BenchmarksBenchmarks are calculated on the openly availableHBN EO/EC taskdataset, consisting of 177,065 files, including metadata.rsbidsis compared topybids,ancpbids-bids, andbids2table. The code for running the benchmarks and generating the figure can be found at thersbids-benchmarkrepository. More information on the method and tasks can be found there.Pybids CompatibilityA compability api can be found underrsbids.pybids. So in general, you can:# replacefrombidsimportBIDSLayout# withfromrsbids.pybidsimportBIDSLayoutAs of now, the indexing and querying methods onBIDSLayoutare implemented with some limitations:BIDSLayout(validate=True)redirects intorsbids.BidsLayout(validate=True), which has a different meaning (validation will eventually be equivalent to pybids, but this needs to be developed)No regex based ignoring of files is possibleBIDSLayoutIndexercan be constructed and used to skip metadata indexing, but all the other fields do nothing.CallingBIDSLayout.get()returns a list ofBIDSPathsas before. The API for this compatibilityBIDSPathis not yet complete (no.copy,.get_associations, or.relpath)Regex searching via.get()is not yet supported.return_type="dir"is also not supportedEntity retrieval methods return a mocked version ofEntity(rsbids has no suchEntityclass). The methods and properties ofEntityare all implemented, however, becausersbidsdoes not use regex when parsing paths, it can only "guess" at thepatternandregexproperties ofEntity. These should not be trusted for any automated use.The methods searching for associated files onBIDSLayoutare not yet implemented (includingget_bval,get_filedmap, etc).get_metadataDOES work.Path building methods and data copying methods are also not implemented (e.g.build_path,write_to_file)database_pathandreset_databaseare both implemented, but usersbidscaches, notpybidsdatabases. So they won't read your previous pybids databases! (Becausersbidsis so fast, caching should not be necessary unless your files are on a network filesystem).That being said, we encourage users to try the new API. Feel free to leave feedback regarding any potential improvements!Notable differences from pybidsAlong with the substantial speed boost,rsbidsoptimizes many aspects of thepybidsapi:Chained queryingrsbids.BidsLayout.get()returns a new instance ofrsbids.BidsLayout. Calls to.getcan thus be chained:view=layout.get(suffix="T1w")# laterview.get(subject="01")Because of this, most of the methods inpybids.BIDSLayoutcan be replaced by an appropriate combination of methods:# pybidslayout.get_subjects(suffix="events",task="stroop")# rsbidslayout.get(suffix="events",task="stroop").entities["subject"]# pybidslayout.get_files(scope="fmriprep")# rsbidslayout.filter(scope="fmriprep")# pybidsforfinlayout.files:...# rsbidsforfinlayout:...Simplified single-file queryingrsbids.BidsLayouthas the.oneproperty, which errors out if the layout does not have exactly one path. If more than one path is present, the entities still to be filtered are listed in the error:# pybids (no error if more than one path)layout.get(subject="001",session="02",suffix="dwi",extension=".nii.gz")[0]# rsbidslayout.get(subject="001",session="02",suffix="dwi",extension=".nii.gz").oneSeperate.get()and.filter()methodspybidsuses the.get()method as an omnibus query method. While convenient, it makes the method brittle because certain arguments are interpreted with special meaning (e.g.scope,target). This makes it challenging to add additional query methods (e.g. searching specificially bypipelineor fileroot).With the split, arguments to.get()will always be interpreted as entity names (e.g.subject,session,run, etc) or metadata keys (e.g.EchoTime, etc). All other special search modes are handled by.filter(). Because each query returns a new layout, it's perfectly possible to chain these calls together, making an extremely flexible query interface..get()accepts the "short" names of entities in addition to their long version. For instance, the following calls are equivalent:layout.get(subject="001")==layout.get(sub=="001").get()also allows you to add a final_to entity names, dropping the_before matching. This is useful for querying python reserved words likefrom:layout.get(from="MNI")# !!! Syntax Errorlayout.get(from_="MNI").filter()currently takes the following arguments:rootRoot searches by dataset root, making it useful for multi-root layouts. It accepts either the complete root as a string, or glob patterns (e.g.**/fmriprep-*).scopeScope uses the same syntax as in pybids:rawandselfboth match the raw dataset,derivativesmatches all derivative datasets,<pipeline_name>searches derivative datasets by pipeline names found in theirdataset_description.json.Note that the above uses ofscopeare primarily included for backward compatibility withpybids. There are (or will be) better, dedicated ways to achieve each of these searches. Moving forward,scopewill be intended to index labelled derivatives (see below).Multi-root layoutspybidssupported single raw or root datasets with multiple, potentially nested derivative datasets.rsbidsreimagines layouts as a flat collection of datasets, each tagged with various attributes. For example, one or more datasets may beraw, and the restderivative. Datasets may be generated with one or morepipelines and derive from one or more datasets. These attributes are (or will be) individually indexed and individually queryable.Thus,rsbidsallows multiple raw roots:# rsbidslayout=rsbids.BidsLayout(["root1","root2"])These roots can be then queried using roots:layout.filter(root="root1")New torsbids, derivatives can be labelled:#rsbidslayout=rsbids.BidsLayout("dataset",derivatives={"proc1":"dataset/derivatives/proc1-v0.10.1","anat":"dataset/derivatives/smriprep-v1.3",})These labels can queried usingscope:layout.filter(scope="anat")All derivatives can be selected using.derivatives:layout.derivatives==layout.filter(scope="derivatives")All datasetrootscan be listed using with:layout.rootsIf the dataset has a single raw root (with any number of derivatives), the.rootattribute can be used to retrieve that root:layout=rsbids.BidsLayout("dataset",derivatives={"proc1":"dataset/derivatives/proc1-v0.10.1","anat":"dataset/derivatives/smriprep-v1.3",})layout.root=="dataset"If there is no raw root, but exactly one derivative root,.rootwill retrieve the derivativelayout=rsbids.BidsLayout("dataset",derivatives={"proc1":"dataset/derivatives/proc1-v0.10.1","anat":"dataset/derivatives/smriprep-v1.3",})layout.filter(scope="proc1").root=="dataset/derivatives/proc1-v0.10.1"All other calls to.rootwill error:layout=rsbids.BidsLayout("dataset",derivatives={"proc1":"dataset/derivatives/proc1-v0.10.1","anat":"dataset/derivatives/smriprep-v1.3",})layout.derivatives.root# !!! Error: multiple rootsThe.descriptionattribute works according to equivalent logic:layout=rsbids.BidsLayout("dataset",derivatives={"proc1":"dataset/derivatives/proc1-v0.10.1","anat":"dataset/derivatives/smriprep-v1.3",})layout.description==<DatasetDescription>Note: The error handling for.descriptionand.rootis still a bit janky.DatasetDescriptionreading has only preliminary support: the object is readonly, and values must be accessed as attributes using snakecase:layout.description.generated_by[0].namelayout.description["Name"]# !!! ErrorMetadata Indexingpybidsdefaults to indexing the metadata, significantly increasing the time to index.rsbidsdefaults to not indexing, since in our experience, the metadata is not needed for most applications. Instead of requesting metadata using an argument on thersbids.BidsLayoutconstructor, metadata is requested using the following method:layout=rsbids.BidsLayout("dataset").index_metadata()This decouples metadata retrieval from layout construction, providing a few advantages:If you discover you later on need metadata, you don't have to reindex the entire layout (especially useful on network-attached filesystems with high latency)You can even index metadata when reading a layout from cacheFunctions consumingBidsLayout(e.g. from 3rd party apps) don't need to worry about whether metadata was indexed or not. If they need metadata, they can simply calllayout.index_metadata(). If metadata is already indexed, the method will immediately returnThe method returns back the same bids layout, so it can be easily chained:layout.index_metadata().get(EchoTime="...")dtypespybidsassociates each entity with a specific datatype. Most entities are strings, but some, such asrun, are explicitely stored as integers.rsbidsstores all entities as strings. This simplifies the layout internals and ensures entities are saved nondestructively. For those used to querying runs with integers, however, fear not!rsbids.get()accepts integer queries for ALL entities:layout.get(subject=1)# will match#   sub-001_T1w.nii.gz#   sub-01_T1w.nii.gz#   sub-1_T1w.nii.gz# but not#   sub-Pre1_T1w.nii.gz#   sub-Treatment001_T1w.nii.gzIf multiple valid matches are found, an error will be thrown.Flexible parsing algorithmrsbidshas two variants of its parsing algorithm. One looks forentity-valuepairs specifically defined by the bids spec (similar to how pybids and all other bids indexers currently work). Invalid entities (..._foobar-val_...) are ignored. This mode is enabled byrsbids.layout(..., validate=True), and gives a validation experiencesomewhatsimilar topybids.BIDSLayout(..., validate=False, is_derivative=True)(note that this will change in the future to match thepybidsdefaults).The other parser is completely generic: it parses any path looking forentity-valuecombinations seperated by underscores (_). So long as the path structure looksroughlybids-like,rsbidsshould correctly parse it, including missing extensions/suffixes, custom entities, any arbitrary value (so long as it has no_), custom datatypes, malformed directory structures, etc.The flexible algorithm currently hasnovalidation, so any path will be parsed intosomethingaccording to the algorithm. In the future,rsbidswill allow for more fine-grained validation.The details of the algorithm will be written at some point in the future. In summary, these are the main priorities:Any valid bids pathMUSTbe parsed correctly (if it's not, it's a bug)Any almost-valid bids pathSHOULDbe parsed correctly. This include paths with one or a few of:Custom entityCustom datatypeCustom entity as a directory if it's also in the file nameFinally, any path bits that can't be interpreted askey-valuepairs will generally be saved asparts(e.g.sub-001_somepart_ses-1_...). In the future,rsbidswill supporting querying for these parts, making it potentially useful even for severely non-bids-compliant datasets.