Repository: bids-standard/pybids

Issue:
Test failure (missing derivatives.json)
Hello, I am looking to package your software for Gentoo Linux.

I am having issues with one test in particular (see end of log):

```
>>> Emerging (1 of 1) sci-libs/pybids-0.4.2::local
 * pybids-0.4.2.tar.gz BLAKE2B SHA512 size ;-) ...                                                       [ ok ]
>>> Unpacking source...
>>> Unpacking pybids-0.4.2.tar.gz to /var/tmp/portage/sci-libs/pybids-0.4.2/work
>>> Source unpacked in /var/tmp/portage/sci-libs/pybids-0.4.2/work
>>> Preparing source in /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2 ...
>>> Source prepared.
>>> Configuring source in /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2 ...
>>> Source configured.
>>> Compiling source in /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2 ...
 * python2_7: running distutils-r1_run_phase distutils-r1_python_compile
/usr/bin/python2.7 setup.py build
running build
running build_py
creating /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python2_7/lib/bids
copying bids/due.py -> /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python2_7/lib/bids
copying bids/version.py -> /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python2_7/lib/bids
copying bids/__init__.py -> /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python2_7/lib/bids
creating /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python2_7/lib/bids/grabbids
copying bids/grabbids/bids_validator.py -> /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python2_7/lib/bids/grabbids
copying bids/grabbids/bids_layout.py -> /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python2_7/lib/bids/grabbids
copying bids/grabbids/__init__.py -> /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python2_7/lib/bids/grabbids
creating /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python2_7/lib/bids/grabbids/config
copying bids/grabbids/config/bids.json -> /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python2_7/lib/bids/grabbids/config
warning: build_py: byte-compiling is disabled, skipping.

 * python3_4: running distutils-r1_run_phase distutils-r1_python_compile
/usr/bin/python3.4 setup.py build
running build
running build_py
creating /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python3_4/lib/bids
copying bids/due.py -> /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python3_4/lib/bids
copying bids/version.py -> /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python3_4/lib/bids
copying bids/__init__.py -> /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python3_4/lib/bids
creating /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python3_4/lib/bids/grabbids
copying bids/grabbids/bids_validator.py -> /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python3_4/lib/bids/grabbids
copying bids/grabbids/bids_layout.py -> /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python3_4/lib/bids/grabbids
copying bids/grabbids/__init__.py -> /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python3_4/lib/bids/grabbids
creating /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python3_4/lib/bids/grabbids/config
copying bids/grabbids/config/bids.json -> /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python3_4/lib/bids/grabbids/config
warning: build_py: byte-compiling is disabled, skipping.

>>> Source compiled.
>>> Test phase: sci-libs/pybids-0.4.2
 * python2_7: running distutils-r1_run_phase python_test
============================================= test session starts ==============================================
platform linux2 -- Python 2.7.14, pytest-3.2.2, py-1.4.34, pluggy-0.4.0 -- /usr/bin/python2.7
cachedir: .cache
rootdir: /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2, inifile:
plugins: cython-0.1.0
collecting ... collected 39 items

bids/grabbids/tests/test_grabbids.py::test_layout_init PASSED
bids/grabbids/tests/test_grabbids.py::test_get_metadata PASSED
bids/grabbids/tests/test_grabbids.py::test_get_metadata2 PASSED
bids/grabbids/tests/test_grabbids.py::test_get_metadata3 PASSED
bids/grabbids/tests/test_grabbids.py::test_get_metadata4 PASSED
bids/grabbids/tests/test_grabbids.py::test_get_events PASSED
bids/grabbids/tests/test_grabbids.py::test_get_events2 PASSED
bids/grabbids/tests/test_grabbids.py::test_get_bvals_bvecs PASSED
bids/grabbids/tests/test_grabbids.py::test_get_subjects PASSED
bids/grabbids/tests/test_grabbids.py::test_get_fieldmap PASSED
bids/grabbids/tests/test_grabbids.py::test_get_fieldmap2 PASSED
bids/grabbids/tests/test_grabbids.py::test_bids_json PASSED
bids/grabbids/tests/test_grabbids.py::test_exclude PASSED
bids/grabbids/tests/test_grabbids.py::test_layout_with_derivs ERROR
bids/grabbids/tests/test_validator.py::test_is_top_level_true PASSED
bids/grabbids/tests/test_validator.py::test_is_top_level_false PASSED
bids/grabbids/tests/test_validator.py::test_is_associated_data_true PASSED
bids/grabbids/tests/test_validator.py::test_is_associated_data_false PASSED
bids/grabbids/tests/test_validator.py::test_is_session_level_true PASSED
bids/grabbids/tests/test_validator.py::test_is_session_level_false PASSED
bids/grabbids/tests/test_validator.py::test_is_subject_level_true PASSED
bids/grabbids/tests/test_validator.py::test_is_subject_false PASSED
bids/grabbids/tests/test_validator.py::test_is_anat_true PASSED
bids/grabbids/tests/test_validator.py::test_is_anat_false PASSED
bids/grabbids/tests/test_validator.py::test_is_dwi_true PASSED
bids/grabbids/tests/test_validator.py::test_is_dwi_false PASSED
bids/grabbids/tests/test_validator.py::test_is_func_true PASSED
bids/grabbids/tests/test_validator.py::test_is_func_false PASSED
bids/grabbids/tests/test_validator.py::test_is_func_bold_true PASSED
bids/grabbids/tests/test_validator.py::test_is_func_bold_false PASSED
bids/grabbids/tests/test_validator.py::test_is_behavioral_true PASSED
bids/grabbids/tests/test_validator.py::test_is_behavioral_false PASSED
bids/grabbids/tests/test_validator.py::test_is_cont_true PASSED
bids/grabbids/tests/test_validator.py::test_is_cont_false PASSED
bids/grabbids/tests/test_validator.py::test_is_field_map_true PASSED
bids/grabbids/tests/test_validator.py::test_is_field_map_false PASSED
bids/grabbids/tests/test_validator.py::test_is_phenotypic_true PASSED
bids/grabbids/tests/test_validator.py::test_is_phenotypic_false PASSED
bids/grabbids/tests/test_validator.py::test_index_associated_false PASSED

==================================================== ERRORS ====================================================
__________________________________ ERROR at setup of test_layout_with_derivs ___________________________________

    @pytest.fixture
    def testlayout3():
        data_dir = join(dirname(__file__), 'data', 'ds005')
>       return BIDSLayout(data_dir, extensions=['derivatives'])

bids/grabbids/tests/test_grabbids.py:25: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python2.7/site-packages/grabbit/core.py:169: in __call__
    return super(LayoutMetaclass, cls).__call__(path, *args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <bids.grabbids.bids_layout.BIDSLayout object at 0x7f82298da490>
path = '/var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2/bids/grabbids/tests/data/ds005'
config = {'entities': [{'directory': '{{root}}/{subject}', 'name': 'subject', 'pattern': '.*sub-([a-zA-Z0-9]+)'}, {'directory':...pattern': '(.*\_scans.tsv)$'}, ...], 'index': {'exclude': ['.*derivatives$', '.*models$', '.*sourcedata$', '.*code$']}}
validate = False, index_associated = True, extensions = ['derivatives'], kwargs = {}
root = '/var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python2_7/lib/bids/grabbids'
fobj = <_io.TextIOWrapper name='/var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python2_7/lib/bids/grabbids/config/bids.json' encoding='UTF-8'>
ext = 'derivatives'

    def __init__(self, path, config=None, validate=False,
                 index_associated=True, extensions=None, **kwargs):
        self.validator = BIDSValidator(index_associated=index_associated)
        self.validate = validate
        if config is None:
            root = dirname(abspath(__file__))
            config = pathjoin(root, 'config', 'bids.json')
    
        # Use dictionary config to allow extension augmentations
        if not isinstance(config, collections.Mapping):
            with open(config) as fobj:
                config = json.load(fobj)
        for ext in extensions or []:
>           with open(pathjoin(root, 'config', '%s.json' % ext)) as fobj:
E           IOError: [Errno 2] No such file or directory: '/var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python2_7/lib/bids/grabbids/config/derivatives.json'

../pybids-0.4.2-python2_7/lib/bids/grabbids/bids_layout.py:32: IOError
====================================== 38 passed, 1 error in 2.27 seconds ======================================
 * ERROR: sci-libs/pybids-0.4.2::local failed (test phase):
 *   (no error message)
 * 
 * Call stack:
 *     ebuild.sh, line  124:  Called src_test
 *   environment, line 2575:  Called distutils-r1_src_test
 *   environment, line  842:  Called _distutils-r1_run_foreach_impl 'python_test'
 *   environment, line  304:  Called python_foreach_impl 'distutils-r1_run_phase' 'python_test'
 *   environment, line 2121:  Called multibuild_foreach_variant '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'python_test'
 *   environment, line 1496:  Called _multibuild_run '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'python_test'
 *   environment, line 1494:  Called _python_multibuild_wrapper 'distutils-r1_run_phase' 'python_test'
 *   environment, line  516:  Called distutils-r1_run_phase 'python_test'
 *   environment, line  773:  Called python_test
 *   environment, line 2477:  Called die
 * The specific snippet of code:
 *       py.test -v || die
 * 
 * If you need support, post the output of `emerge --info '=sci-libs/pybids-0.4.2::local'`,
 * the complete build log and the output of `emerge -pqv '=sci-libs/pybids-0.4.2::local'`.
 * The complete build log is located at '/var/tmp/portage/sci-libs/pybids-0.4.2/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/sci-libs/pybids-0.4.2/temp/environment'.
 * Working directory: '/var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2'
 * S: '/var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2'

>>> Failed to emerge sci-libs/pybids-0.4.2, Log file:

>>>  '/var/tmp/portage/sci-libs/pybids-0.4.2/temp/build.log'

 * Messages for package sci-libs/pybids-0.4.2:

 * ERROR: sci-libs/pybids-0.4.2::local failed (test phase):
 *   (no error message)
 * 
 * Call stack:
 *     ebuild.sh, line  124:  Called src_test
 *   environment, line 2575:  Called distutils-r1_src_test
 *   environment, line  842:  Called _distutils-r1_run_foreach_impl 'python_test'
 *   environment, line  304:  Called python_foreach_impl 'distutils-r1_run_phase' 'python_test'
 *   environment, line 2121:  Called multibuild_foreach_variant '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'python_test'
 *   environment, line 1496:  Called _multibuild_run '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'python_test'
 *   environment, line 1494:  Called _python_multibuild_wrapper 'distutils-r1_run_phase' 'python_test'
 *   environment, line  516:  Called distutils-r1_run_phase 'python_test'
 *   environment, line  773:  Called python_test
 *   environment, line 2477:  Called die
 * The specific snippet of code:
 *       py.test -v || die
 * 
 * If you need support, post the output of `emerge --info '=sci-libs/pybids-0.4.2::local'`,
 * the complete build log and the output of `emerge -pqv '=sci-libs/pybids-0.4.2::local'`.
 * The complete build log is located at '/var/tmp/portage/sci-libs/pybids-0.4.2/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/sci-libs/pybids-0.4.2/temp/environment'.
 * Working directory: '/var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2'
 * S: '/var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2'
 * 
 * The following package has failed to build, install, or execute postinst:
 * 
 *  (sci-libs/pybids-0.4.2:0/0::local, ebuild scheduled for merge), Log file:
 *   '/var/tmp/portage/sci-libs/pybids-0.4.2/temp/build.log'
 * 
quiethost ~ # ls /var/tmp/portage/sci-libs/pybids-0.4.2/work/pybids-0.4.2-python2_7/lib/bids/grabbids/config/ -lah
total 12K
drwxr-xr-x 2 portage portage 4.0K Mar  5 00:48 .
drwxr-xr-x 3 portage portage 4.0K Mar  5 00:48 ..
-rw-r--r-- 1 portage portage 1.7K Nov 17 02:27 bids.json
```

The directory seems to have been correctly installed in the sandbox, and one of the files inside it is also present, just that the other file, is not installed. Apparently the release is broken, as the tests require the file, but [the file is not selected to be installed](https://github.com/INCF/pybids/blob/37f0bf3f7ce3a1bf6fb31376b8249c324f572048/setup.py#L26). I notice this has been fixed in the meantime. Perhaps you could release a patch?


Discussion:
This should be addressed when #135 is merged, which should be later today or tomorrow. Thanks!
Will you be making a new release? otherwise a new commit on master will do little to address the viability of your `0.4.2` release. Though Gentoo has excellent support for live (i.e. latest commit) software, most people prefer statically versioned packages.
Yes, there will be new releases of both grabbit and pybids.

Resolving patch:
diff --git a/bids/version.py b/bids/version.py
--- a/bids/version.py
+++ b/bids/version.py
@@ -1,4 +1,5 @@
 from __future__ import absolute_import, division, print_function
+import os
 
 # Format expected by setup.py and doc/source/conf.py: string of form "X.Y.Z"
 _version_major = 0
@@ -64,8 +65,24 @@
 MICRO = _version_micro
 VERSION = __version__
 # No data for now
-# PACKAGE_DATA = {'bids': [pjoin('data', '*')]}
 REQUIRES = ["grabbit>=0.1.1", "six", "num2words"]
 EXTRAS_REQUIRE = {
     'analysis': ['numpy', 'scipy', 'pandas', 'nibabel', 'patsy'],
 }
+TESTS_REQUIRE = ["pytest>=3.3.0"]
+
+
+def package_files(directory):
+    # from https://stackoverflow.com/questions/27664504/how-to-add-package-data-recursively-in-python-setup-py
+    paths = []
+    for (path, directories, filenames) in os.walk(directory):
+        for filename in filenames:
+            paths.append(os.path.join('..', path, filename))
+    return paths
+
+extra_files = package_files('path_to/extra_files_dir')
+PACKAGE_DATA = {
+    'bids.grabbids': ['config/*.json'],
+    'bids.reports': ['config/*.json'],
+    'bids': package_files('bids/tests/data')
+}
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -23,9 +23,10 @@
             platforms=PLATFORMS,
             version=VERSION,
             packages=PACKAGES,
-            package_data={'bids': ['grabbids/config/*.json']},
+            package_data=PACKAGE_DATA,
             install_requires=REQUIRES,
-            extras_require=EXTRAS_REQUIRE)
+            extras_require=EXTRAS_REQUIRE,
+            tests_require=TESTS_REQUIRE)
 
 
 if __name__ == '__main__':


Test patch: