Repository: bids-standard/pybids

Issue:
Layout does not follow symbolic links to directories in pybids>=0.17.0
In 0.16.5, I can use symbolic links to create a virtual dataset for a single subject

```bash
virtualDataset=/tmp/virtualDataset
ln -s /data/realDataset/dataset_description.json ${virtualDataset}
ln -s /data/realDataset/sub-01 ${virtualDataset}
```

Then in Python I can do

```python
import bids
bids.BIDSLayout('/tmp/virtualDataset')
```

and it will show the subject and sessions, and I can run layout.get, everything seems to work normally.

With pybids==0.17.1, the layout is empty. However, it only has a problem with directory symlinks, it works if I link the files, eg I can do

```bash
virtualDataset=/tmp/virtualDataset
ln -s /data/realDataset/dataset_description.json ${virtualDataset}
mkdir -p ${virtualDataset}/sub-01/ses-MR1/anat
ln -s /data/realDataset/sub-01/ses-MR1/anat/* ${virtualDataset}/sub-01/ses-MR1/anat
```

I've reproduced this on Linux and Mac, with Python 3.11 and 3.9 respectively.

EDIT: narrowed down the range of versions: 0.16.5 reads the symlinked data, 0.17.0 does not

Discussion:
@cookpa: I think I might have found the issue

https://github.com/bids-standard/pybids/blob/95c7aa942d982685a0f22daa3951b4ccc692bdd6/bids/layout/index.py#L208C9-L208C63

This calls [fsspec.spec.walk](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.walk), which says

```
Note that the “files” outputted will include anything that is not a directory, such as links.
```

If I can fix this by detecting directories, I'll open a PR.







Solution patch:
diff --git a/bids/layout/index.py b/bids/layout/index.py
index 31663b2d..8f163228 100644
--- a/bids/layout/index.py
+++ b/bids/layout/index.py
@@ -207,6 +207,21 @@ def _index_dir(self, path, config, force=None):
         # Get lists of 1st-level subdirectories and files in the path directory
         _, dirnames, filenames = next(path.fs.walk(path.path))
 
+        # Symbolic links are returned as filenames even if they point to directories
+        # Temporary list to store symbolic links that need to be removed from filenames
+        links_to_dirs = []
+
+        # Find directories in filenames
+        for possible_link in filenames:
+            full_path = path / possible_link
+            if full_path.is_dir():
+                links_to_dirs.append(possible_link)
+
+        dirnames.extend(links_to_dirs)
+
+        for link in links_to_dirs:
+            filenames.remove(link)
+
         # Move any directories suffixed with .zarr to filenames
         zarrnames = [entry for entry in dirnames if Path(entry).suffix == '.zarr']
         dirnames = [entry for entry in dirnames if Path(entry).suffix != '.zarr']
@@ -332,7 +347,7 @@ def load_json(path):
 
                 to_store = (file_ents, payload, bf.path)
                 file_data[key][bf._dirname].append(to_store)
-                
+
         # To avoid integrity errors, track primary keys we've seen
         seen_assocs = set()
 
@@ -494,7 +509,7 @@ def create_association_pair(src, dst, kind, kind2=None):
                     self.session.add(all_entities[md_key])
                 tag = _create_tag_dict(bf, all_entities[md_key], md_val, is_metadata=True)
                 all_tag_dicts.append(tag)
-                
+
         self.session.bulk_save_objects(all_objs)
         self.session.bulk_insert_mappings(Tag, all_tag_dicts)
         self.session.commit()


Test patch:
diff --git a/bids/layout/tests/test_layout.py b/bids/layout/tests/test_layout.py
index 664fdd04..1cc86758 100644
--- a/bids/layout/tests/test_layout.py
+++ b/bids/layout/tests/test_layout.py
@@ -1140,3 +1140,22 @@ def test_indexer_patterns(fname):
         [re.compile(r"/\.")],
         root=root,
     ) is (".datalad" in fname)
+
+
+def test_symlinks_in_path(tmp_path):
+
+    src_ds = os.path.abspath(join(get_test_data_path(), '7t_trt'))
+    src_sub = join(src_ds, 'sub-04')
+
+    # This will be a normal directory with symlink contents
+    ds_with_links = Path(tmp_path / '7t_trt').absolute()
+
+    os.makedirs(ds_with_links, exist_ok=False)
+
+    link_ds_description = ds_with_links / 'dataset_description.json'
+    link_sub = ds_with_links / 'sub-04'
+
+    os.symlink(join(src_ds, 'dataset_description.json'), link_ds_description)
+    os.symlink(src_sub, link_sub)
+
+    assert "Subjects: 1 | Sessions: 2 | Runs: 2" in str(BIDSLayout(tmp_path / "7t_trt"))