Repository: datalad/datalad-neuroimaging (language: python)

Issue:
Crippled file system test failure
This was the last time it ran successfully: https://github.com/datalad/datalad-neuroimaging/actions/runs/2320161731
Upgrade `bids.py` metadata extractor
It would be useful if the bids extractor (and for some points, eventually all other extractors in this extension) could:
- be compatible with the new generation of metadata handling, i.e.:
   - inherit metadata classes from datalad-metalad (not `datalad.metadata.*`)
   - make use of the distinction between dataset-level and file-level metadata extraction
- extract everything there is to extract from a datalad dataset (i.e. not from annexed data) **before** `datalad get`ting any file content that might be necessary for further extraction. (currently the extraction starts by getting all required file content...)
- be compatible with, and use updated functionality of, the latest stable version of pybids

I've made a start at this. I'm working on this within the context of the catalog: likely many of our future users will be working with BIDS data and would want to extract BIDS metadata and have it rendered in the catalog. So I have an idea of the BIDS-related metadata that would be useful in the catalog, but I'm keen to get input from other @datalad/developers if there are features that you think will be useful to include.


Discussion / hints:

Also tagging @surchs and @CPernet based on likely shared interest.
I Wholeheartedly support this initiative! 

FWIW, Regarding getting content: for use/integration with datalad-fuse I am thinking of adding some config variable to metalad to bypass getting content by metalad since it would be available (only needed portions of the file) via fuse. Might need though a mode which would first query datalad-fuse on either it can access that file data, and get only when it can't (eg file on some fancy special remote datalad-fuse has no clue of how to access via fsspec)
Oh wow, didn't know about [datalad-fuse](https://github.com/datalad/datalad-fuse) and now I do, great! I'm guessing that functionality (check whether possible to get partial content via fuse, otherwise `datalad get`) would probably live in metalad? The extension extractor could possibly specify whether it needs access to no / partial / full file content.
> I'm guessing that functionality (check whether possible to get partial content via fuse, otherwise `datalad get`) would probably live in metalad?

May be ;-) in principle it could be a mode of operation on datalad-fuse , so that is it fails to access file via fsspec, it would just get it in full. Then metalad wouldn't need to deal with that.

> The extension extractor could possibly specify whether it needs access to no / partial / full file content.

I thought about the same but still hope we could avoid that.
Here's a sample of output from the [BOLD5000 dataset](https://github.com/OpenNeuroDatasets/ds001499) that I ran locally:

```
datalad clone https://github.com/OpenNeuroDatasets/ds001499
cd ds001499
datalad -f json_pp meta-extract -d . bids_dataset
```

yields:

```
{
  "action": "meta_extract",
  "metadata_record": {
    "agent_email": "s.heunis@fz-juelich.de",
    "agent_name": "Stephan Heunis",
    "dataset_id": "3e874376-b053-11e8-b9ac-0242ac130026",
    "dataset_version": "5be66b27ab5e033e9163caa94cec882bd4cee1d0",
    "extracted_metadata": {
      "@context": {
        "age(years)": {
          "@id": "pato:0000011",
          "description": "age of a sample (organism) at the time of data acquisition in years",
          "unit": "uo:0000036",
          "unit_label": "year"
        },
        "bids": {
          "@id": "http://bids.neuroimaging.io/bids_spec1.0.2.pdf#",
          "description": "ad-hoc vocabulary for the Brain Imaging Data Structure (BIDS) standard",
          "type": "http://purl.org/dc/dcam/VocabularyEncodingScheme"
        }
      },
      "Acknowledgements": "We thank Scott Kurdilla for his patience as our MRI technologist throughout all data collection. We would also like to thank Austin Marcus for his assistance in various stages of this project, Jayanth Koushik for his assistance in AlexNet feature extractions, and Ana Van Gulick for her assistance with public data distribution and open science issues. Finally, we thank our participants for their participation and patience, without them this dataset would not have been possible.",
      "BIDSVersion": "1.0.2",
      "DatasetDOI": "10.18112/openneuro.ds001499.v1.3.1",
      "HowToAcknowledge": "Please cite our paper available on arXiv: http://arxiv.org/abs/1809.01281",
      "author": [
        "Nadine Chang",
        "John A. Pyles",
        "Austin Marcus",
        "Abhinav Gupta",
        "Michael J. Tarr",
        "Elissa M. Aminoff"
      ],
      "citation": [
        "https://bold5000.org"
      ],
      "conformsto": "http://bids.neuroimaging.io/bids_spec1.0.2.pdf",
      "entities": {
        "acquisition": [
          "spinecho",
          "spinechopf68",
          "PA",
          "AP"
        ],
        "datatype": [
          "anat",
          "func",
          "fmap",
          "dwi"
        ],
        "direction": [
          "PA",
          "AP"
        ],
        "extension": [
          ".tsv",
          ".bval",
          ".nii.gz",
          ".tsv.gz",
          ".bvec",
          ".json"
        ],
        "fmap": [
          "epi"
        ],
        "recording": [
          "cardiac",
          "respiratory",
          "trigger"
        ],
        "run": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10
        ],
        "session": [
          "12",
          "15",
          "04",
          "06",
          "13",
          "14",
          "10",
          "03",
          "08",
          "05",
          "11",
          "07",
          "02",
          "01",
          "09",
          "16"
        ],
        "subject": [
          "CSI1",
          "CSI4",
          "CSI2",
          "CSI3"
        ],
        "suffix": [
          "events",
          "participants",
          "sessions",
          "T2w",
          "physio",
          "T1w",
          "epi",
          "dwi",
          "bold",
          "description"
        ],
        "task": [
          "localizer",
          "5000scenes"
        ]
      },
      "fundedby": "This dataset was collected with the support of NSF Award BCS-1439237 to Elissa M. Aminoff and Michael J. Tarr, ONR MURI N000141612007 and Sloan, Okawa Fellowship to Abhinav Gupta, and NSF Award BSC-1640681 to Michael Tarr.",
      "license": "CC0",
      "name": "BOLD5000",
      "readme": "BOLD5000: Brains, Objects, Landscapes Dataset\n\nFor details please refer to BOLD5000.org and our paper on arXiv (http://arxiv.org/abs/1809.01281)\n\n*Participant Directories Content*\n1) Four participants: CSI1, CSI2, CSI3, & CSI4\n2) Functional task data acquisition sessions: sessions #1-15\nEach functional session includes:\n-3 sets of fieldmaps (EPI opposite phase encoding; spin-echo opposite phase encoding pairs with partial & non-partial Fourier)\n-9 or 10 functional scans of slow event-related 5000 scene data (5000scenes)\n-1 or 0 functional localizer scans used to define scene selective regions (localizer)\n-each event.json file lists each stimulus, the onset time, and the participant\u2019s response (participants performed a simple valence task) \n3) Anatomical data acquisition session: #16\nAnatomical Data: T1 weighted MPRAGE scan, a T2 weighted SPACE, diffusion spectrum imaging   \n\nNotes:\n-All MRI and fMRI data provided is with Siemens pre-scan normalization filter.  \n-CSI4 only participated in 10 MRI sessions: 1-9 were functional acquisition sessions, and 10 was the anatomical data acquisition session.\n\n*Derivatives Directory Content*\n1) fMRIprep: \n-Preprocessed data for all functional data of CSI1 through CSI4 (listed in folders for each participant: derivatives/fmriprep/sub-CSIX). Data was preprocessed both in T1w image space and on surface space. Functional data was motion corrected, susceptibility distortion corrected, and aligned to the anatomical data using bbregister. Please refer to the paper for the details on preprocessing.\n-Reports resulting from fMRI prep, which include the success of anatomical alignment and distortion correction, among other measures of preprocessing success are all listed in the sub-CSIX.html files.  \n2) Freesurfer: Freesurfer reconstructions as a result of fMRIprep preprocessing stream. \n3) MRIQC: Image quality metrics (IQMs) of the dataset using MRIQC. \n-CSIX-func.csv files are text files with a list of all IQMs for each session, for each run.\n-CSIX-anat.csv files are text files with a list of all IQMs for the scans acquired in the anatomical session (e.g., MPRAGE). \n-CSIX_IQM.xls an excel workbook, each sheet of workbook lists the IQMs for a single run. This is the same data as CSIX-func.csv, except formatted differently. \n-sub-CSIX/derivatives: contain .json with the MRIQC/IQM results for each run. \n-sub-CSIX/reports: contains .html file with MRIQC/IQM results for each run along with mean signal and standard deviation maps. \n4)spm: A directory that contains the masks used to define each region of interest (ROI) in each participant. There were 10 ROIs: early visual (EarlyVis), lateral occipital cortex (LOC), occipital place area (OPA), parahippocampal place area (PPA), retrosplenial complex (RSC) for the left hemisphere (LH) and right hemisphere (RH).",
      "variables": {
        "dataset": [
          "subject",
          "age",
          "handedness",
          "sex",
          "suffix"
        ],
        "subject": [
          "session",
          "subject",
          "At this point in the day, you have eaten...",
          "Date",
          "Did you work out today?",
          "Do you drink alcoholic beverages?",
          "Do you drink caffeinated beverages (i.e. coffee, tea, coke, etc.)?",
          "Do you smoke?",
          "Duration (in seconds)",
          "End Date",
          "Have you taken ibuprofen today (e.g. Advil, Motrin)?",
          "How long ago was your last meal?",
          "How many hours of sleep did you get last night?",
          "If so, what was the activity, and how long ago?",
          "If so, when was the last time you had a caffeinated beverage?",
          "If so, when was the last time you had an alcoholic beverage?",
          "If so, when was the last time you smoked?",
          "If so, when was the last time you took it?",
          "In the scanner today I was mentally: - Click to write Choice 1",
          "In the scanner today I was physically: - Click to write Choice 1",
          "Is there anything you think we should know about your experience in the MRI (e.g. were you tired, confused about task instructions, etc)?",
          "Is this...",
          "Is this....1",
          "Is this....2",
          "Is this....3",
          "Is this....4",
          "Is this....5",
          "Please comment on which of these things, if any, are particularly different today and/or if you think they have affected your performance (for better or for worse):",
          "Progress",
          "Recorded Date",
          "Response ID",
          "Session",
          "Start Date",
          "Subject ID",
          "Time",
          "suffix"
        ]
      }
    },
    "extraction_parameter": {},
    "extraction_time": 1643231572.23382,
    "extractor_name": "bids_dataset",
    "extractor_version": "0.0.1",
    "type": "dataset"
  },
  "path": "/Users/jsheunis/Documents/psyinf/Data/ds001499",
  "status": "ok",
  "type": "dataset"
}
```

It contains all the fields extracted by the existing bids extractor, and additionally:
- `entities`: the results of `bids.BIDSLayout.get_[entity]` (where entity could be `subject`, `run`, etc). 
- `variables`: the results of `bids.BIDSLayout.get_collections(level=[entity])[0].to_df().columns` (where entity is one of `run`, `session`, `subject`, `dataset`). The variable collection from `entity='dataset'` gives the column headings of the `participants.tsv` file. The variable collection from `entity='run'` will only yield a result if annexed content is available locally.
- `readme`: populated from the readme.md file, if available; _not_ replacing the `description` field.

The extra info from `entities` and `variables` could all be useful tags on the dataset level, especially when filtering/quering.

Lastly, the above all runs on the light-weight datalad dataset, i.e. does not require local access to annexed content (assuming a `text2git` configuration).

Fix patch:
diff --git a/datalad_neuroimaging/extractors/bids_dataset.py b/datalad_neuroimaging/extractors/bids_dataset.py
new file mode 100644
index 0000000..abf40e5
--- /dev/null
+++ b/datalad_neuroimaging/extractors/bids_dataset.py
@@ -0,0 +1,315 @@
+# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
+# ex: set sts=4 ts=4 sw=4 noet:
+# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
+#
+#   See COPYING file distributed along with the datalad package for the
+#   copyright and license terms.
+#
+# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
+"""Metadata extractor for BIDS dataset-level information"""
+import logging
+from pathlib import Path
+from uuid import UUID
+
+from bids import BIDSLayout
+from datalad.log import log_progress
+from datalad.metadata.definitions import vocabulary_id
+from datalad.utils import ensure_unicode
+from datalad_metalad.extractors.base import (
+    DataOutputCategory,
+    DatasetMetadataExtractor,
+    ExtractorResult,
+)
+
+lgr = logging.getLogger("datalad.metadata.extractors.bids_dataset")
+
+
+class BIDSProperties:
+    """Main properties used in BIDS v1.6.0 dataset description"""
+    NAME = "Name"
+    BIDSVERSION = "BIDSVersion"
+    DATASETTYPE = "DatasetType"
+    LICENSE = "License"
+    AUTHORS = "Authors"
+    ACKNOWLEDGEMENTS = "Acknowledgements"
+    HOWTOACKNOWLEDGE = "HowToAcknowledge"
+    FUNDING = "Funding"
+    ETHICSAPPROVALS = "EthicsApprovals"
+    REFSANDLINKS = "ReferencesAndLinks"
+    DATASETDOI = "DatasetDOI"
+    HEDVERSION = "HEDVersion"
+    README = "ReadMe"
+
+
+class BIDSEntities:
+    """Main entities used in BIDS v1.6.0 filenames"""
+    SUBJECT = "subject"
+    SESSION = "session"
+    TASK = "task"
+    ACQUISITION = "acquisition"
+    CEAGENT = "ceagent"
+    RECONSTRUCTION = "reconstruction"
+    DIRECTION = "direction"
+    RUN = "run"
+    PROC = "proc"
+    MODALITY = "modality"
+    ECHO = "echo"
+    FLIP = "flip"
+    INV = "inv"
+    MT = "mt"
+    PART = "part"
+    RECORDING = "recording"
+    SPACE = "space"
+    SUFFIX = "suffix"
+    SCANS = "scans"
+    FMAP = "fmap"
+    DATATYPE = "datatype"
+    EXTENSION = "extension"
+
+
+BIDSCONTEXT = {
+    "@id": "https://doi.org/10.5281/zenodo.4710751",
+    "description": "ad-hoc vocabulary for the Brain Imaging Data Structure (BIDS) standard v1.6.0",
+    "type": vocabulary_id,
+}
+
+REQUIRED_BIDS_FILES = ["dataset_description.json", "participants.tsv"]
+
+DATASET = "dataset"
+
+# The following mappings are one-to-one currently, but could change in future
+BIDS_PROPERTIES_MAPPING = {
+    BIDSProperties.NAME: BIDSProperties.NAME,
+    BIDSProperties.BIDSVERSION: BIDSProperties.BIDSVERSION,
+    BIDSProperties.DATASETTYPE: BIDSProperties.DATASETTYPE,
+    BIDSProperties.LICENSE: BIDSProperties.LICENSE,
+    BIDSProperties.AUTHORS: BIDSProperties.AUTHORS,
+    BIDSProperties.ACKNOWLEDGEMENTS: BIDSProperties.ACKNOWLEDGEMENTS,
+    BIDSProperties.HOWTOACKNOWLEDGE: BIDSProperties.HOWTOACKNOWLEDGE,
+    BIDSProperties.FUNDING: BIDSProperties.FUNDING,
+    BIDSProperties.ETHICSAPPROVALS: BIDSProperties.ETHICSAPPROVALS,
+    BIDSProperties.REFSANDLINKS: BIDSProperties.REFSANDLINKS,
+    BIDSProperties.DATASETDOI: BIDSProperties.DATASETDOI,
+    BIDSProperties.HEDVERSION: BIDSProperties.HEDVERSION,
+}
+
+BIDS_ENTITIES_MAPPING = {
+    BIDSEntities.SUBJECT: BIDSEntities.SUBJECT,
+    BIDSEntities.SESSION: BIDSEntities.SESSION,
+    BIDSEntities.TASK: BIDSEntities.TASK,
+    BIDSEntities.ACQUISITION: BIDSEntities.ACQUISITION,
+    BIDSEntities.CEAGENT: BIDSEntities.CEAGENT,
+    BIDSEntities.RECONSTRUCTION: BIDSEntities.RECONSTRUCTION,
+    BIDSEntities.DIRECTION: BIDSEntities.DIRECTION,
+    BIDSEntities.RUN: BIDSEntities.RUN,
+    BIDSEntities.PROC: BIDSEntities.PROC,
+    BIDSEntities.MODALITY: BIDSEntities.MODALITY,
+    BIDSEntities.ECHO: BIDSEntities.ECHO,
+    BIDSEntities.FLIP: BIDSEntities.FLIP,
+    BIDSEntities.INV: BIDSEntities.INV,
+    BIDSEntities.MT: BIDSEntities.MT,
+    BIDSEntities.PART: BIDSEntities.PART,
+    BIDSEntities.RECORDING: BIDSEntities.RECORDING,
+    BIDSEntities.SPACE: BIDSEntities.SPACE,
+    BIDSEntities.SUFFIX: BIDSEntities.SUFFIX,
+    BIDSEntities.SCANS: BIDSEntities.SCANS,
+    BIDSEntities.FMAP: BIDSEntities.FMAP,
+    BIDSEntities.DATATYPE: BIDSEntities.DATATYPE,
+    BIDSEntities.EXTENSION: BIDSEntities.EXTENSION,
+}
+
+# Entities for which variable collections can be extracted without requiring
+# file content
+BIDS_COLLECTION_ENTITIES = [
+    BIDSEntities.RUN,
+    BIDSEntities.SESSION,
+    BIDSEntities.SUBJECT,
+    DATASET,
+]
+
+
+class BIDSDatasetExtractor(DatasetMetadataExtractor):
+    """
+    Top level BIDS extractor class interfacing with metalad
+    Inherits from metalad's DatasetMetadataExtractor class
+    """
+
+    def get_id(self) -> UUID:
+        # Generated using V4 UUID: https://www.uuidgenerator.net/
+        return UUID("a05726b7-86e0-408c-a0ef-08f3d85df47b")
+
+    def get_version(self) -> str:
+        return "0.0.1"
+
+    def get_data_output_category(self) -> DataOutputCategory:
+        return DataOutputCategory.IMMEDIATE
+
+    def get_required_content(self):
+        # TODO: logging
+        bids_dir = _find_bids_root(self.dataset.path)
+
+        for filename in REQUIRED_BIDS_FILES:
+            rslt = self.dataset.get(
+                bids_dir / filename, on_failure="ignore", return_type="list"
+            )
+            if "status" in rslt[0] and rslt[0]["status"] == "impossible":
+                msg = (f"The file '{filename}' should be part of the BIDS "
+                "dataset in order for the 'bids_dataset' extractor to function "
+                "correctly")
+                raise FileNotFoundError(msg)
+        return True
+
+    def extract(self, _=None) -> ExtractorResult:
+        log_progress(
+            lgr.info,
+            "extractorsbidsdataset",
+            f"Start bids_dataset metadata extraction from {self.dataset.path}",
+            total=2,
+            label="bids_dataset metadata extraction",
+            unit=" Dataset",
+        )
+        return ExtractorResult(
+            extractor_version=self.get_version(),
+            extraction_parameter=self.parameter or {},
+            extraction_success=True,
+            datalad_result_dict={"type": "dataset", "status": "ok"},
+            immediate_data=BIDSmeta(self.dataset).get_metadata(),
+        )
+
+
+class BIDSmeta(object):
+    """
+    The extractor class for BIDS dataset-level metadata
+
+    Extracts dataset-level metadata of a BIDS-compliant dataset, using
+    PyBIDS
+    """
+
+    def __init__(self, dataset) -> None:
+        self.dataset = dataset
+
+    def get_metadata(self):
+        """
+        Function to load BIDSLayout and trigger metadata extraction
+        """
+        bids_dir = _find_bids_root(self.dataset.path)
+        # Check if derivatives are in BIDS dataset
+        deriv_dir = bids_dir / "derivatives"
+        derivative_exist = deriv_dir.exists()
+        # TODO: handle case with amoty or nonexisting derivatives directory
+        # TODO: decide what to do with meta_data from derivatives, if anything
+        # Call BIDSLayout with dataset path and derivatives boolean
+        bids = BIDSLayout(bids_dir, derivatives=derivative_exist)
+        dsmeta = self._get_dsmeta(bids)
+        log_progress(
+            lgr.info,
+            "extractorsbidsdataset",
+            f"Finished bids_dataset metadata extraction from {bids_dir}",
+        )
+        return dsmeta
+
+    def _get_dsmeta(self, bids):
+        """
+        Internal function to extract metadata from pyBIDS' BIDSLayout
+        
+        This function executes the following steps:
+        1. Extract metadata from `dataset_description.json`
+        2. Extract README text
+        3. Extract information about entities
+        4. Extract variable collection information on multiple levels
+           (dataset, subject, session, run). The dataset level
+           collection will grab variables from participants.tsv
+        5. Add context to metadata output
+        """
+        # STEP 1: Extract metadata from `dataset_description.json`
+        metadata = self._get_bids_dsdescription(bids)
+        # STEP 2: Extract README text
+        metadata["description"] = self._get_bids_readme()
+        # STEP 3: Extract information about entities and add to metadata
+        metadata["entities"] = self._get_bids_entities(bids)
+        # STEP 4: Extract variable collection information on multiple levels
+        metadata["variables"] = self._get_bids_variables(bids)
+        # STEP 5: Add context to metadata output
+        metadata["@context"] = BIDSCONTEXT
+        return metadata
+
+    def _get_bids_dsdescription(self, bids):
+        """Get BIDS dataset description"""
+        # Get info from 'dataset_description.json'
+        dsdesc_dict = bids.get_dataset_description()
+        # Map extracted dict keys to standard keys
+        return {BIDS_PROPERTIES_MAPPING.get(k, k): v for k, v in dsdesc_dict.items()}
+
+    def _get_bids_readme(self):
+        """Get text from README, if any"""
+        readme = []
+        # Grab all readme files, loop through
+        for README_fname in [
+            file for file in Path(self.dataset.path).glob("[Rr][Ee][Aa][Dd][Mm][Ee]*")
+        ]:
+            # datalad get content if annexed
+            self.dataset.get(README_fname)
+            # read text from file
+            try:
+                file_text = ensure_unicode(README_fname.read_text()).strip()
+            except:
+                file_text = ""
+            # Append dict with file text + extension to list
+            readme.append({"extension": README_fname.suffix, "text": file_text})
+        return readme if readme else None
+
+    def _get_bids_entities(self, bids):
+        """Get dataset-specific entities from BIDSLayout"""
+        ds_entities = list(bids.entities.keys())
+        new_entities = {}
+        # If the entity is in the main list AND in the list
+        # created from the dataset (i.e. in BIDSLayout),
+        # the entity values can be retrieved using
+        # BIDSLayout.get_[entity]()
+        for ent in BIDS_ENTITIES_MAPPING.keys():
+            if ent in ds_entities:
+                class_method = getattr(bids, f"get_{ent}")
+                new_entities[ent] = class_method()
+                # Delete key/value if value is empty list
+                if not new_entities[ent]:
+                    del new_entities[ent]
+        return new_entities
+
+    def _get_bids_variables(self, bids):
+        """Get variable collection information from BIDSLayout"""
+        # Extract variable collection information on multiple levels
+        # levels (dataset, subject, session, run). The dataset level
+        # collection will grab variables from participants.tsv
+        variables = {}
+        for ent in BIDS_COLLECTION_ENTITIES:
+            try:
+                cols = bids.get_collections(level=ent)
+                if cols:
+                    variables[ent] = list(cols[0].to_df().columns)
+            except Exception as e:
+                # 'run' collection would require file content, i.e.
+                # it will (intentionally) fail on a dataset-level
+                pass
+        return variables
+
+
+def _find_bids_root(dataset_path) -> Path:
+    """
+    Find relative location of BIDS directory within datalad dataset
+    """
+    participant_paths = list(Path(dataset_path).glob("**/participants.tsv"))
+    # 1 - if more than one, select first and output warning
+    # 2 - if zero, output error
+    # 3 - if 1, add to dataset path and set ats bids root dir
+    if len(participant_paths) == 0:
+        msg = ("The file 'participants.tsv' should be part of the BIDS dataset "
+        "in order for the 'bids_dataset' extractor to function correctly")
+        raise FileNotFoundError(msg)
+    elif len(participant_paths) > 1:
+        msg = (f"Multiple 'participants.tsv' files ({len(participant_paths)}) "
+        f"were found in the recursive filetree of {dataset_path}, selecting "
+        "first path.")
+        lgr.warning(msg)
+        return Path(participant_paths[0]).parent
+    else:
+        return Path(participant_paths[0]).parent
diff --git a/docs/source/conf.py b/docs/source/conf.py
index d6fdb23..aa1cb41 100644
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -120,7 +120,7 @@ def setup(sphinx):
 #
 # This is also used if you do content translation via gettext catalogs.
 # Usually you set "language" from the command line for these cases.
-language = None
+language = 'en'
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
diff --git a/requirements-devel.txt b/requirements-devel.txt
index 5db3241..1b30680 100644
--- a/requirements-devel.txt
+++ b/requirements-devel.txt
@@ -3,3 +3,4 @@ nose
 coverage
 sphinx
 sphinx_rtd_theme
+pytest
diff --git a/setup.cfg b/setup.cfg
index 512b8bf..585bf03 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -14,13 +14,14 @@ classifiers =
 
 [options]
 zip_safe = False
-python_requires = >= 3.5
+python_requires = >= 3.7
 install_requires =
-    datalad >= 0.12.0
+    datalad == 0.16.7
     pydicom  # DICOM metadata
-    pybids >= 0.9.2  # BIDS metadata
+    pybids >= 0.15.1  # BIDS metadata
     nibabel  # NIfTI metadata
     pandas  # bids2scidata export
+    datalad-metalad >= 0.3.1
 packages = find:
 include_package_data = True
 
@@ -60,6 +61,7 @@ datalad.metadata.extractors =
     dicom = datalad_neuroimaging.extractors.dicom:MetadataExtractor
     nidm = datalad_neuroimaging.extractors.nidm:MetadataExtractor
     nifti1 = datalad_neuroimaging.extractors.nifti1:MetadataExtractor
+    bids_dataset = datalad_neuroimaging.extractors.bids_dataset:BIDSDatasetExtractor
 datalad.tests =
     neuroimaging = datalad_neuroimaging
 


Test patch:
diff --git a/.github/workflows/test_crippledfs.yml b/.github/workflows/test_crippledfs.yml
index 717b822..4057946 100644
--- a/.github/workflows/test_crippledfs.yml
+++ b/.github/workflows/test_crippledfs.yml
@@ -28,10 +28,10 @@ jobs:
         git config --global user.email "test@github.land"
         git config --global user.name "GitHub Almighty"
     - uses: actions/checkout@v1
-    - name: Set up Python 3.6
+    - name: Set up Python 3.7
       uses: actions/setup-python@v1
       with:
-        python-version: 3.6
+        python-version: 3.7
     - name: Install dependencies
       run: |
         pip install -r requirements-devel.txt
diff --git a/datalad_neuroimaging/extractors/tests/test_bids_dataset.py b/datalad_neuroimaging/extractors/tests/test_bids_dataset.py
new file mode 100644
index 0000000..eb92aae
--- /dev/null
+++ b/datalad_neuroimaging/extractors/tests/test_bids_dataset.py
@@ -0,0 +1,161 @@
+# emacs: -*- mode: python-mode; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil; coding: utf-8 -*-
+# ex: set sts=4 ts=4 sw=4 noet:
+# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
+#
+#   See COPYING file distributed along with the datalad package for the
+#   copyright and license terms.
+#
+# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
+"""Test BIDS-dataset metadata extractor """
+
+from math import isnan
+from os.path import join as opj
+from simplejson import dumps
+from datalad.api import Dataset
+
+from nose.tools import assert_equal
+from datalad.support.external_versions import external_versions
+from datalad.tests.utils import with_tree
+from datalad.tests.utils import assert_in
+from datalad.tests.utils import known_failure_osx
+from datalad.tests.utils import known_failure_windows
+
+
+from datalad.tests.utils import skip_if_no_module
+skip_if_no_module('bids')
+
+from datalad_neuroimaging.extractors.bids_dataset import BIDSmeta
+from unittest import TestCase
+TestCase.maxDiff = None
+
+bids_template = {
+    '.datalad': {
+        'config': '[datalad "metadata"]\n  nativetype = bids',},
+    'dataset_description.json': """
+{
+    "Name": "The mother of all experiments",
+    "BIDSVersion": "1.4.0",
+    "DatasetType": "raw",
+    "License": "CC0",
+    "Authors": [
+        "Paul Broca",
+        "Carl Wernicke"
+    ],
+    "Acknowledgements": "Special thanks to Korbinian Brodmann for help in formatting this dataset in BIDS. We thank Alan Lloyd Hodgkin and Andrew Huxley for helpful comments and discussions about the experiment and manuscript; Hermann Ludwig Helmholtz for administrative support; and Claudius Galenus for providing data for the medial-to-lateral index analysis.",
+    "HowToAcknowledge": "Please cite this paper: https://www.ncbi.nlm.nih.gov/pubmed/001012092119281",
+    "Funding": [
+        "National Institute of Neuroscience Grant F378236MFH1",
+        "National Institute of Neuroscience Grant 5RMZ0023106"
+    ],
+    "EthicsApprovals": [
+        "Army Human Research Protections Office (Protocol ARL-20098-10051, ARL 12-040, and ARL 12-041)"
+    ],
+    "ReferencesAndLinks": [
+        "https://www.ncbi.nlm.nih.gov/pubmed/001012092119281",
+        "Alzheimer A., & Kraepelin, E. (2015). Neural correlates of presenile dementia in humans. Journal of Neuroscientific Data, 2, 234001. doi:1920.8/jndata.2015.7"
+    ],
+    "DatasetDOI": "doi:10.0.2.3/dfjj.10",
+    "HEDVersion": "7.1.1"
+}
+""",
+    'participants.tsv': u"""\
+participant_id\tgender\tage\thandedness\thearing_problems_current\tlanguage
+sub-01\tn/a\t30-35\tr\tn\tрусский
+sub-03\tf\t20-25\tr\tn\tenglish
+""",
+    'README.md': """
+A very detailed description с юникодом
+""",
+    'sub-01': {'func': {'sub-01_task-some_bold.nii.gz': ''}},
+    'sub-03': {'func': {'sub-03_task-other_bold.nii.gz': ''}},
+    'derivatives': {'empty': {}}  # Test that we do not blow if derivatives present
+}
+
+correct_metadata = """\
+{
+    "Acknowledgements": "Special thanks to Korbinian Brodmann for help in formatting this dataset in BIDS. We thank Alan Lloyd Hodgkin and Andrew Huxley for helpful comments and discussions about the experiment and manuscript; Hermann Ludwig Helmholtz for administrative support; and Claudius Galenus for providing data for the medial-to-lateral index analysis.",
+    "Authors": [
+        "Carl Wernicke",
+        "Paul Broca"
+    ],
+    "BIDSVersion": "1.4.0",
+    "DatasetDOI": "doi:10.0.2.3/dfjj.10",
+    "DatasetType": "raw",
+    "EthicsApprovals": [
+        "Army Human Research Protections Office (Protocol ARL-20098-10051, ARL 12-040, and ARL 12-041)"
+    ],
+    "Funding": [
+        "National Institute of Neuroscience Grant 5RMZ0023106",
+        "National Institute of Neuroscience Grant F378236MFH1"
+    ],
+    "HEDVersion": "7.1.1",
+    "HowToAcknowledge": "Please cite this paper: https://www.ncbi.nlm.nih.gov/pubmed/001012092119281",
+    "License": "CC0",
+    "Name": "The mother of all experiments",
+    "ReferencesAndLinks": [
+        "Alzheimer A., & Kraepelin, E. (2015). Neural correlates of presenile dementia in humans. Journal of Neuroscientific Data, 2, 234001. doi:1920.8/jndata.2015.7",
+        "https://www.ncbi.nlm.nih.gov/pubmed/001012092119281"
+    ],
+    "description": [
+        {
+            "extension": ".md",
+            "text": "A very detailed description с юникодом"
+        }
+    ],
+    "entities": {
+        "datatype": [
+            "func"
+        ],
+        "extension": [
+            ".json",
+            ".md",
+            ".nii.gz",
+            ".tsv"
+        ],
+        "subject": [
+            "01",
+            "03"
+        ],
+        "suffix": [
+            "README",
+            "bold",
+            "description",
+            "participants"
+        ],
+        "task": [
+            "other",
+            "some"
+        ]
+    },
+    "variables": {
+        "dataset": [
+            "age",
+            "gender",
+            "handedness",
+            "hearing_problems_current",
+            "language",
+            "subject",
+            "suffix"
+        ]
+    }
+}"""
+
+@known_failure_windows
+@with_tree(tree=bids_template)
+def test_get_metadata(path):
+    ds = Dataset(path).create(force=True)
+    meta = BIDSmeta(ds).get_metadata()
+    del meta['@context']
+    sort_lists_in_dict(meta)
+    dump = dumps(meta, sort_keys=True, indent=4, ensure_ascii=False)
+    assert_equal(
+        dump,
+        correct_metadata
+    )
+
+def sort_lists_in_dict(data_dict):
+    for key, value in data_dict.items():
+        if isinstance(value, list):
+            data_dict[key].sort()
+        if isinstance(value, dict):
+            sort_lists_in_dict(data_dict[key])
\ No newline at end of file