Repository: mila-udem/fuel (python)

Issue:
fuel-download / fuel-convert for datasets not in the fuel distro
I have written a fairly comprehensive fuel adapter for the [Labeled Faces in the Wild dataset](http://vis-www.cs.umass.edu/lfw/). However, instead of asking to have this code merged into the fuel library, I'm asking that `fuel-download` and `fuel-convert` be extended to allow conversion of datasets that are not part of the fuel distribution itself.

I've put together a proof of concept for this. In my [fuel_lfw](https://github.com/dribnet/lfw_fuel) repo is [lfw.py](https://github.com/dribnet/lfw_fuel/blob/857dfb43bbcdeb5d820c95b541569af181617409/lfw_fuel/lfw.py) which includes a [`download_subparser` hook](https://github.com/dribnet/lfw_fuel/blob/857dfb43bbcdeb5d820c95b541569af181617409/lfw_fuel/lfw.py#L58) for fuel-download and a [`convert_subparser` hook](https://github.com/dribnet/lfw_fuel/blob/857dfb43bbcdeb5d820c95b541569af181617409/lfw_fuel/lfw.py#L172) for fuel-convert. Since `fuel-download` and `fuel-convert` are currently not currently intended for this use case I've also built my own short wrapper scripts called [`kero-download`](https://github.com/dribnet/kerosene/blob/23a480903519a3da1b2d379ebaba6004adf2566a/bin/kero-download) and [`kero-convert`](https://github.com/dribnet/kerosene/blob/23a480903519a3da1b2d379ebaba6004adf2566a/bin/kero-convert), which trick those scripts into working correctly. So using those, I can simply execute:

```bash
kero-download lfw_fuel/lfw.py
kero-convert lfw_fuel/lfw.py
```

to produce a fuel hdf5 file ready to be consumed by my blocks program.

In addition, the I provide an [`H5PYDataset` subclass](https://github.com/dribnet/lfw_fuel/blob/857dfb43bbcdeb5d820c95b541569af181617409/lfw_fuel/lfw.py#L182) so that other developers can easily use this dataset in their own blocks programs.

Separate from LFW, I have a handful of datasets that I would like to make available in a similar fashion.

So I'm hoping to get:
  * agreement from the fuel maintainers that this is a useful use-case to allow
  * functionality added to `fuel-download` and `fuel-convert` to support use-case (I can also help here)
  * some guidance on what the conventions should be for these 3rd party fuel datasets

Hopefully the fuel maintainers agree that this change makes fuel more powerful by allowing anyone to create and distribute datasets for the fuel ecosystem. 

Pull request description:
[WIP] Allow for external downloaders/converters
Fixes #213.

@dribnet the interface for adding external downloaders / converters would be to define an `extra_downloaders` / `extra_converters` config attribute in `~/.fuelrc` like so:

```
# ~/.fuelrc

extra_downloaders: ['my.extra.downloaders1', 'my.extra.downloaders2']
extra_converters: ['my.extra.converters1', 'my.extra.converters2']
```

where elements of `extra_downloaders` / `extra_converters` are modules which define an `all_downloaders` / `all_converters` attribute like in `fuel.downloaders` / `fuel.converters`.

Would that work for you?

Patch:
diff --git a/docs/built_in_datasets.rst b/docs/built_in_datasets.rst
index 41168e9..1f3e459 100644
--- a/docs/built_in_datasets.rst
+++ b/docs/built_in_datasets.rst
@@ -13,8 +13,25 @@ to automate these operations.
 Environment variable
 --------------------
 
-In order for Fuel to know where to look for its data, the ``FUEL_DATA_PATH``
-environment variable has to be set.
+In order for Fuel to know where to look for its data, the ``data_path``
+configuration variable has to be set inside ``~/.fuelrc``. It's expected to be
+a sequence of paths separated by an OS-specific delimiter (``:`` for Linux and
+OSX, ``;`` for Windows):
+
+.. code-block:: yaml
+
+    # ~/.fuelrc
+    data_path: "/first/path/to/my/data:/second/path/to/my/data"
+
+When looking for a specific file (e.g. ``mnist.hdf5``), Fuel will search each of
+these paths in sequence, using the first matching file that it finds.
+
+This configuration variable can be overridden by setting the ``FUEL_DATA_PATH``
+environment variable:
+
+.. code-block:: bash
+
+    $ export FUEL_DATA_PATH="/first/path/to/my/data:/second/path/to/my/data"
 
 Let's now change directory for the rest of this tutorial:
 
@@ -124,3 +141,47 @@ argument:
 
         H5PYDataset     0.1
         fuel.converters 0.1
+
+
+Working with external packages
+------------------------------
+
+By default, Fuel looks for downloaders and converters in the
+``fuel.downloaders`` and ``fuel.converters`` modules, respectively, but you're
+not limited to that.
+
+Fuel can be told to look into additional modules by setting the
+``extra_downloaders`` and ``extra_converters`` configuration variables in
+``~/.fuelrc``. These variables are expected to be lists of module names.
+
+For instance, suppose you'd like to include the following modules:
+
+* ``package1.extra_downloaders``
+* ``package2.extra_downloaders``
+* ``package1.extra_converters``
+* ``package2.extra_converters``
+
+You should include the following in your ``~/.fuelrc``:
+
+.. code-block:: yaml
+
+    # ~/.fuelrc
+    extra_downloaders:
+    - package1.extra_downloaders
+    - package2.extra_downloaders
+    extra_converters:
+    - package1.extra_converters
+    - package2.extra_converters
+
+These configuration variables can be overridden through the
+``FUEL_EXTRA_DOWNLOADERS`` and ``FUEL_EXTRA_CONVERTERS`` environment variables,
+which are expected to be strings of space-separated module names, like so:
+
+.. code-block:: bash
+
+    export FUEL_EXTRA_DOWNLOADERS="package1.extra_downloaders package2.extra_downloaders"
+    export FUEL_EXTRA_CONVERTERS="package1.extra_converters package2.extra_converters"
+
+This feature lets external developers define their own Fuel dataset
+downloader/converter packages, and also makes working with private datasets more
+straightforward.
diff --git a/docs/new_dataset.rst b/docs/new_dataset.rst
index 6337112..b4ab7a9 100644
--- a/docs/new_dataset.rst
+++ b/docs/new_dataset.rst
@@ -44,9 +44,8 @@ accessible to ``fuel-download``, they need to include it in the
 ``all_downloaders`` attribute of the ``fuel.downloaders`` subpackage.
 
 The function accepts an :class:`argparse.ArgumentParser` instance as input and
-should set a function as the default value for the ``func`` argument of the
-parser. Put the following piece of code inside the ``fuel.downloaders.iris``
-module (you'll have to create it):
+should return a downloading function. Put the following piece of code inside
+the ``fuel.downloaders.iris`` module (you'll have to create it):
 
 .. code-block:: python
 
@@ -54,10 +53,10 @@ module (you'll have to create it):
 
     def fill_subparser(subparser):
         subparser.set_defaults(
-            func=default_downloader,
             urls=['https://archive.ics.uci.edu/ml/machine-learning-databases/'
                   'iris/iris.data'],
             filenames=['iris.data'])
+        return default_downloader
 
 You should also register Iris as a downloadable dataset via the
 ``all_downloaders`` attribute. It's a tuple of pairs of name and subparser
@@ -82,16 +81,15 @@ subparser to fill. Users will then be able to type the ``fuel-download iris``
 command and ``iris.data`` will be downloaded.
 
 When the ``fuel-download iris`` command is typed, the download script will call
-the function passed as the ``func`` argument and give it the
+the function returned by ``fill_subparser`` and give it the
 :class:`argparse.Namespace` instance containing all parsed command line
 arguments. That function is responsible for downloading the data.
 
-This is why we called the ``set_defaults`` method: it allowed us to define which
-function would get called to download our data. We used the
-:meth:`~.downloaders.base.default_downloader` convenience function. It expects
-the parsed arguments to contain a list of URLs and a list of filenames,
-and downloads each URL, saving it under its corresponding filename. This is why
-we also included the ``urls`` and ``filenames`` default arguments.
+We used the :meth:`~.downloaders.base.default_downloader` convenience function
+as our download function. It expects the parsed arguments to contain a list of
+URLs and a list of filenames, and downloads each URL, saving it under its
+corresponding filename. This is why we set the ``urls`` and ``filenames``
+default arguments.
 
 If your use case is more exotic, you can just as well define your own download
 function. Be aware of the following default arguments:
@@ -167,7 +165,7 @@ module (you'll have to create it):
         return (output_path,)
 
     def fill_subparser(subparser):
-        subparser.set_defaults(func=convert_iris)
+        return convert_iris
 
 We used the convenience :meth:`~.converters.base.fill_hdf5_file` function
 to populate our HDF5 file and create the split array. This function expects
@@ -190,8 +188,8 @@ Here's an example of how the init file might look:
     from fuel.converters import iris
 
     all_converters = (
-        ('binarized_mnist', binarized_mnist.convert_binarized_mnist),
-        ('iris', iris.convert_iris))
+        ('binarized_mnist', binarized_mnist.fill_subparser),
+        ('iris', iris.fill_subparser))
 
 Dataset subclass
 ----------------
@@ -250,19 +248,18 @@ You can now use the Iris dataset like you would use any other built-in dataset:
     >>> from fuel.downloaders.base import default_downloader
     >>> def fill_downloader_subparser(subparser):
     ...     subparser.set_defaults(
-    ...         func=default_downloader,
     ...         urls=['https://archive.ics.uci.edu/ml/machine-learning-databases/'
     ...               'iris/iris.data'],
     ...         filenames=['iris.data'])
+    ...     return default_downloader
     >>> import argparse
     >>> parser = argparse.ArgumentParser()
     >>> __ = parser.add_argument("--directory", type=str, default=os.getcwd())
     >>> __ = parser.add_argument("--clear", action='store_true')
     >>> subparsers = parser.add_subparsers()
-    >>> fill_downloader_subparser(subparsers.add_parser('iris'))
+    >>> download_function = fill_downloader_subparser(subparsers.add_parser('iris'))
     >>> args = parser.parse_args(['iris'])
     >>> args_dict = vars(args)
-    >>> func = args_dict.pop('func')
     >>> content = b''
     >>> for i in range(50):
     ...    content += b'0.0,0.0,0.0,0.0,Iris-setosa\n'
@@ -281,7 +278,7 @@ You can now use the Iris dataset like you would use any other built-in dataset:
     ...     mock_response.headers = {'content-length': length}
     ...     mock_requests.get.return_value = mock_response
     ...     func(**args_dict)
-    >>> call_download(func, args_dict) # doctest: +ELLIPSIS
+    >>> call_download(download_function, args_dict) # doctest: +ELLIPSIS
     Downloading ...
 
 .. doctest::
@@ -320,17 +317,16 @@ You can now use the Iris dataset like you would use any other built-in dataset:
     ...     h5file.close()
     ...     return (output_path,)
     >>> def fill_converter_subparser(subparser):
-    ...     subparser.set_defaults(func=iris_converter)
+    ...     return iris_converter
     >>> parser = argparse.ArgumentParser()
     >>> __ = parser.add_argument("--directory", type=str, default=os.getcwd())
     >>> __ = parser.add_argument("--output-directory", type=str,
     ...                          default=os.getcwd())
     >>> subparsers = parser.add_subparsers()
-    >>> fill_converter_subparser(subparsers.add_parser('iris'))
+    >>> convert_function = fill_converter_subparser(subparsers.add_parser('iris'))
     >>> args = parser.parse_args(['iris'])
     >>> args_dict = vars(args)
-    >>> func = args_dict.pop('func')
-    >>> output_paths = func(**args_dict)
+    >>> output_paths = convert_function(**args_dict)
     >>> os.remove('iris.data')
 
 .. doctest::
@@ -363,5 +359,78 @@ You can now use the Iris dataset like you would use any other built-in dataset:
     
     >>> os.remove('iris.hdf5')
 
+Working with external packages
+------------------------------
+
+To distribute Fuel-compatible downloaders and converters independently from
+Fuel, you have to write your own modules or subpackages which will define
+``all_downloaders`` and ``all_converters``. Here is how the Iris downloader
+and converter might look like if you were to include them as part of the
+``my_fuel`` package:
+
+.. code-block:: python
+
+    # my_fuel/downloaders/iris_downloader.py
+    from fuel.downloaders.base import default_downloader
+
+    def fill_subparser(subparser):
+        subparser.set_defaults(
+            urls=['https://archive.ics.uci.edu/ml/machine-learning-databases/'
+                  'iris/iris.data'],
+            filenames=['iris.data'])
+        return default_downloader
+
+.. code-block:: python
+
+    # my_fuel/downloaders/__init__.py
+    from my_fuel.downloaders import iris
+
+    all_downloaders = (('iris', iris.fill_subparser),)
+
+.. code-block:: python
+
+    # my_fuel/converters/iris.py
+    import os
+
+    import h5py
+    import numpy
+
+    from fuel.converters.base import fill_hdf5_file
+
+
+    def convert_iris(directory, output_directory, output_filename='iris.hdf5'):
+        output_path = os.path.join(output_directory, output_filename)
+        h5file = h5py.File(output_path, mode='w')
+        classes = {'Iris-setosa': 0, 'Iris-versicolor': 1, 'Iris-virginica': 2}
+        # ...
+
+    def fill_subparser(subparser):
+        return convert_iris
+
+.. code-block:: python
+
+    # my_fuel/converters/__init__.py
+    from my_fuel.converters import iris
+
+    all_converters = (('iris', iris.fill_subparser),)
+
+In order to use the downloaders and converters defined in ``my_fuel``, users
+would then have to set the ``extra_downloaders`` and ``extra_converters``
+configuration variables inside ``~/.fuelrc`` like so:
+
+.. code-block:: yaml
+
+    extra_downloaders: ['my_fuel.downloaders']
+    extra_converters: ['my_fuel.converters']
+
+or set the ``FUEL_EXTRA_DOWNLOADERS`` and ``FUEL_EXTRA_CONVERTERS`` environment
+variables like so (this would override the ``extra_downloaders`` and
+``extra_converters`` configuration variables):
+
+.. code-block:: bash
+
+    $ export FUEL_EXTRA_DOWNLOADERS=my_fuel.downloaders
+    $ export FUEL_EXTRA_CONVERTERS=my_fuel.converters
+
 .. _Iris dataset: https://archive.ics.uci.edu/ml/datasets/Iris
 .. _iris.data: https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data
diff --git a/fuel/bin/fuel_convert.py b/fuel/bin/fuel_convert.py
index 858e941..31f9f40 100755
--- a/fuel/bin/fuel_convert.py
+++ b/fuel/bin/fuel_convert.py
@@ -1,11 +1,13 @@
 #!/usr/bin/env python
 """Fuel dataset conversion utility."""
 import argparse
+import importlib
 import os
 import sys
 
 import h5py
 
+import fuel
 from fuel import converters
 from fuel.converters.base import MissingInputFiles
 from fuel.datasets import H5PYDataset
@@ -33,6 +35,14 @@ def main(args=None):
 
     """
     built_in_datasets = dict(converters.all_converters)
+    if fuel.config.extra_converters:
+        for name in fuel.config.extra_converters:
+            extra_datasets = dict(
+                importlib.import_module(name).all_converters)
+            if any(key in built_in_datasets for key in extra_datasets.keys()):
+                raise ValueError('extra converters conflict in name with '
+                                 'built-in converters')
+            built_in_datasets.update(extra_datasets)
     parser = argparse.ArgumentParser(
         description='Conversion script for built-in datasets.')
     subparsers = parser.add_subparsers()
@@ -40,24 +50,23 @@ def main(args=None):
     parent_parser.add_argument(
         "-d", "--directory", help="directory in which input files reside",
         type=str, default=os.getcwd())
-    for name, subparser_fn in built_in_datasets.items():
+    convert_functions = {}
+    for name, fill_subparser in built_in_datasets.items():
         subparser = subparsers.add_parser(
             name, parents=[parent_parser],
             help='Convert the {} dataset'.format(name))
         subparser.add_argument(
             "-o", "--output-directory", help="where to save the dataset",
             type=str, default=os.getcwd(), action=CheckDirectoryAction)
-        subparser_fn(subparser)
+        # Allows the parser to know which subparser was called.
+        subparser.set_defaults(which_=name)
+        convert_functions[name] = fill_subparser(subparser)
 
     args = parser.parse_args(args)
     args_dict = vars(args)
+    convert_function = convert_functions[args_dict.pop('which_')]
     try:
-        func = args_dict.pop('func')
-    except KeyError:
-        parser.print_usage()
-        parser.exit()
-    try:
-        output_paths = func(**args_dict)
+        output_paths = convert_function(**args_dict)
     except MissingInputFiles as e:
         intro = "The following required files were not found:\n"
         message = "\n".join([intro] + ["   * " + f for f in e.filenames])
diff --git a/fuel/bin/fuel_download.py b/fuel/bin/fuel_download.py
index 194b784..16cd2af 100755
--- a/fuel/bin/fuel_download.py
+++ b/fuel/bin/fuel_download.py
@@ -1,8 +1,10 @@
 #!/usr/bin/env python
 """Fuel dataset downloading utility."""
 import argparse
+import importlib
 import os
 
+import fuel
 from fuel import downloaders
 from fuel.downloaders.base import NeedURLPrefix
 
@@ -28,6 +30,14 @@ def main(args=None):
 
     """
     built_in_datasets = dict(downloaders.all_downloaders)
+    if fuel.config.extra_downloaders:
+        for name in fuel.config.extra_downloaders:
+            extra_datasets = dict(
+                importlib.import_module(name).all_downloaders)
+            if any(key in built_in_datasets for key in extra_datasets.keys()):
+                raise ValueError('extra downloaders conflict in name with '
+                                 'built-in downloaders')
+            built_in_datasets.update(extra_datasets)
     parser = argparse.ArgumentParser(
         description='Download script for built-in datasets.')
     parent_parser = argparse.ArgumentParser(add_help=False)
@@ -37,19 +47,19 @@ def main(args=None):
     parent_parser.add_argument(
         "--clear", help="clear the downloaded files", action='store_true')
     subparsers = parser.add_subparsers()
-    for name, subparser_fn in built_in_datasets.items():
-        subparser_fn(subparsers.add_parser(
+    download_functions = {}
+    for name, fill_subparser in built_in_datasets.items():
+        subparser = subparsers.add_parser(
             name, parents=[parent_parser],
-            help='Download the {} dataset'.format(name)))
+            help='Download the {} dataset'.format(name))
+        # Allows the parser to know which subparser was called.
+        subparser.set_defaults(which_=name)
+        download_functions[name] = fill_subparser(subparser)
     args = parser.parse_args()
     args_dict = vars(args)
+    download_function = download_functions[args_dict.pop('which_')]
     try:
-        func = args_dict.pop('func')
-    except KeyError:
-        parser.print_usage()
-        parser.exit()
-    try:
-        func(**args_dict)
+        download_function(**args_dict)
     except NeedURLPrefix:
         parser.error(url_prefix_message)
 
diff --git a/fuel/config_parser.py b/fuel/config_parser.py
index d0509b7..04738c1 100644
--- a/fuel/config_parser.py
+++ b/fuel/config_parser.py
@@ -51,6 +51,18 @@ The following configurations are supported:
    The default :class:`~numpy.dtype` to use for floating point numbers. The
    default value is ``float64``. A lower value can save memory.
 
+.. option:: extra_downloaders
+
+   A list of package names which, like fuel.downloaders, define an
+   `all_downloaders` attribute listing available downloaders. By default,
+   an empty list.
+
+.. option:: extra_converters
+
+   A list of package names which, like fuel.converters, define an
+   `all_converters` attribute listing available converters. By default,
+   an empty list.
+
 .. _YAML: http://yaml.org/
 .. _environment variables:
    https://en.wikipedia.org/wiki/Environment_variable
@@ -59,6 +71,7 @@ The following configurations are supported:
 import logging
 import os
 
+import six
 import yaml
 
 from .exceptions import ConfigurationError
@@ -68,6 +81,21 @@ logger = logging.getLogger(__name__)
 NOT_SET = object()
 
 
+def extra_downloader_converter(value):
+    """Parses extra_{downloader,converter} arguments.
+
+    Parameters
+    ----------
+    value : iterable or str
+        If the value is a string, it is split into a list using spaces
+        as delimiters. Otherwise, it is returned as is.
+
+    """
+    if isinstance(value, six.string_types):
+        value = value.split(" ")
+    return value
+
+
 class Configuration(object):
     def __init__(self):
         self.config = {}
@@ -144,6 +172,10 @@ config = Configuration()
 # Define configuration options
 config.add_config('data_path', type_=str, env_var='FUEL_DATA_PATH')
 config.add_config('default_seed', type_=int, default=1)
+config.add_config('extra_downloaders', type_=extra_downloader_converter,
+                  default=[], env_var='FUEL_EXTRA_DOWNLOADERS')
+config.add_config('extra_converters', type_=extra_downloader_converter,
+                  default=[], env_var='FUEL_EXTRA_CONVERTERS')
 
 # Default to Theano's floatX if possible
 try:
diff --git a/fuel/converters/binarized_mnist.py b/fuel/converters/binarized_mnist.py
index 7a77292..eb00e78 100644
--- a/fuel/converters/binarized_mnist.py
+++ b/fuel/converters/binarized_mnist.py
@@ -83,4 +83,4 @@ def fill_subparser(subparser):
         Subparser handling the `binarized_mnist` command.
 
     """
-    subparser.set_defaults(func=convert_binarized_mnist)
+    return convert_binarized_mnist
diff --git a/fuel/converters/caltech101_silhouettes.py b/fuel/converters/caltech101_silhouettes.py
index c8b8328..91e5328 100644
--- a/fuel/converters/caltech101_silhouettes.py
+++ b/fuel/converters/caltech101_silhouettes.py
@@ -73,4 +73,4 @@ def fill_subparser(subparser):
     subparser.add_argument(
         "size", type=int, choices=(16, 28),
         help="height/width of the datapoints")
-    subparser.set_defaults(func=convert_silhouettes)
+    return convert_silhouettes
diff --git a/fuel/converters/cifar10.py b/fuel/converters/cifar10.py
index 9aaa177..3d30338 100644
--- a/fuel/converters/cifar10.py
+++ b/fuel/converters/cifar10.py
@@ -106,4 +106,4 @@ def fill_subparser(subparser):
         Subparser handling the `cifar10` command.
 
     """
-    subparser.set_defaults(func=convert_cifar10)
+    return convert_cifar10
diff --git a/fuel/converters/cifar100.py b/fuel/converters/cifar100.py
index 23bcbc1..1c28e18 100644
--- a/fuel/converters/cifar100.py
+++ b/fuel/converters/cifar100.py
@@ -104,4 +104,4 @@ def fill_subparser(subparser):
         Subparser handling the `cifar100` command.
 
     """
-    subparser.set_defaults(func=convert_cifar100)
+    return convert_cifar100
diff --git a/fuel/converters/iris.py b/fuel/converters/iris.py
index a79ae4e..e5758a8 100644
--- a/fuel/converters/iris.py
+++ b/fuel/converters/iris.py
@@ -55,4 +55,12 @@ def convert_iris(directory, output_directory, output_filename='iris.hdf5'):
 
 
 def fill_subparser(subparser):
-    subparser.set_defaults(func=convert_iris)
+    """Sets up a subparser to convert the Iris dataset file.
+
+    Parameters
+    ----------
+    subparser : :class:`argparse.ArgumentParser`
+        Subparser handling the `iris` command.
+
+    """
+    return convert_iris
diff --git a/fuel/converters/mnist.py b/fuel/converters/mnist.py
index 4f4d3ab..d0f0cd7 100644
--- a/fuel/converters/mnist.py
+++ b/fuel/converters/mnist.py
@@ -105,7 +105,7 @@ def fill_subparser(subparser):
         "--dtype", help="dtype to save to; by default, images will be " +
         "returned in their original unsigned byte format",
         choices=('float32', 'float64', 'bool'), type=str, default=None)
-    subparser.set_defaults(func=convert_mnist)
+    return convert_mnist
 
 
 def read_mnist_images(filename, dtype=None):
diff --git a/fuel/converters/svhn.py b/fuel/converters/svhn.py
index 1aa9e5d..b63d54c 100644
--- a/fuel/converters/svhn.py
+++ b/fuel/converters/svhn.py
@@ -376,4 +376,4 @@ def fill_subparser(subparser):
     """
     subparser.add_argument(
         "which_format", help="which dataset format", type=int, choices=(1, 2))
-    subparser.set_defaults(func=convert_svhn)
+    return convert_svhn
diff --git a/fuel/downloaders/binarized_mnist.py b/fuel/downloaders/binarized_mnist.py
index 70149f3..706f9ce 100644
--- a/fuel/downloaders/binarized_mnist.py
+++ b/fuel/downloaders/binarized_mnist.py
@@ -21,5 +21,5 @@ def fill_subparser(subparser):
     urls = ['http://www.cs.toronto.edu/~larocheh/public/datasets/' +
             'binarized_mnist/binarized_mnist_{}.amat'.format(s) for s in sets]
     filenames = ['binarized_mnist_{}.amat'.format(s) for s in sets]
-    subparser.set_defaults(
-        func=default_downloader, urls=urls, filenames=filenames)
+    subparser.set_defaults(urls=urls, filenames=filenames)
+    return default_downloader
diff --git a/fuel/downloaders/caltech101_silhouettes.py b/fuel/downloaders/caltech101_silhouettes.py
index 6a6e069..3fdfa6d 100644
--- a/fuel/downloaders/caltech101_silhouettes.py
+++ b/fuel/downloaders/caltech101_silhouettes.py
@@ -34,5 +34,4 @@ def fill_subparser(subparser):
     subparser.add_argument(
         "size", type=int, choices=(16, 28),
         help="height/width of the datapoints")
-    subparser.set_defaults(
-        func=silhouettes_downloader)
+    return silhouettes_downloader
diff --git a/fuel/downloaders/cifar10.py b/fuel/downloaders/cifar10.py
index a6cb848..d61a4a5 100644
--- a/fuel/downloaders/cifar10.py
+++ b/fuel/downloaders/cifar10.py
@@ -17,5 +17,5 @@ def fill_subparser(subparser):
     """
     url = 'http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz'
     filename = 'cifar-10-python.tar.gz'
-    subparser.set_defaults(
-        func=default_downloader, urls=[url], filenames=[filename])
+    subparser.set_defaults(urls=[url], filenames=[filename])
+    return default_downloader
diff --git a/fuel/downloaders/cifar100.py b/fuel/downloaders/cifar100.py
index b5c5123..b5fe2b2 100644
--- a/fuel/downloaders/cifar100.py
+++ b/fuel/downloaders/cifar100.py
@@ -17,5 +17,5 @@ def fill_subparser(subparser):
     """
     url = 'http://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz'
     filename = 'cifar-100-python.tar.gz'
-    subparser.set_defaults(func=default_downloader, urls=[url],
-                           filenames=[filename])
+    subparser.set_defaults(urls=[url], filenames=[filename])
+    return default_downloader
diff --git a/fuel/downloaders/iris.py b/fuel/downloaders/iris.py
index 63cd6e6..9716fc3 100644
--- a/fuel/downloaders/iris.py
+++ b/fuel/downloaders/iris.py
@@ -16,7 +16,7 @@ def fill_subparser(subparser):
 
     """
     subparser.set_defaults(
-        func=default_downloader,
         urls=['https://archive.ics.uci.edu/ml/machine-learning-databases/'
               'iris/iris.data'],
         filenames=['iris.data'])
+    return default_downloader
diff --git a/fuel/downloaders/mnist.py b/fuel/downloaders/mnist.py
index c3da4d2..8bafc28 100644
--- a/fuel/downloaders/mnist.py
+++ b/fuel/downloaders/mnist.py
@@ -20,5 +20,5 @@ def fill_subparser(subparser):
     filenames = ['train-images-idx3-ubyte.gz', 'train-labels-idx1-ubyte.gz',
                  't10k-images-idx3-ubyte.gz', 't10k-labels-idx1-ubyte.gz']
     urls = ['http://yann.lecun.com/exdb/mnist/' + f for f in filenames]
-    subparser.set_defaults(
-        func=default_downloader, urls=urls, filenames=filenames)
+    subparser.set_defaults(urls=urls, filenames=filenames)
+    return default_downloader
diff --git a/fuel/downloaders/svhn.py b/fuel/downloaders/svhn.py
index d7ab7ca..961b39d 100644
--- a/fuel/downloaders/svhn.py
+++ b/fuel/downloaders/svhn.py
@@ -28,4 +28,4 @@ def fill_subparser(subparser):
     """
     subparser.add_argument(
         "which_format", help="which dataset format", type=int, choices=(1, 2))
-    subparser.set_defaults(func=svhn_downloader)
+    return svhn_downloader


Test patch:
diff --git a/tests/test_config_parser.py b/tests/test_config_parser.py
index b649d4d..d69e9b8 100644
--- a/tests/test_config_parser.py
+++ b/tests/test_config_parser.py
@@ -1,9 +1,23 @@
 import os
 import tempfile
 
-from numpy.testing import assert_raises
+from numpy.testing import assert_equal, assert_raises
 
-from fuel.config_parser import Configuration, ConfigurationError
+from fuel.config_parser import (Configuration, ConfigurationError,
+                                extra_downloader_converter)
+
+
+class TestExtraDownloaderConverter(object):
+    def test_iterable(self):
+        assert_equal(extra_downloader_converter(['a.b.c', 'd.e.f']),
+                     ['a.b.c', 'd.e.f'])
+
+    def test_str(self):
+        assert_equal(extra_downloader_converter("a.b.c d.e.f"),
+                     ['a.b.c', 'd.e.f'])
+
+    def test_str_one_element(self):
+        assert_equal(extra_downloader_converter("a.b.c"), ['a.b.c'])
 
 
 def test_config_parser():
diff --git a/tests/test_converters.py b/tests/test_converters.py
index b013577..1e48cad 100644
--- a/tests/test_converters.py
+++ b/tests/test_converters.py
@@ -141,11 +141,10 @@ class TestMNIST(object):
         subparser.set_defaults(
             directory=self.tempdir, output_directory=self.tempdir,
             output_filename='mock_mnist.hdf5')
-        mnist.fill_subparser(subparser)
+        convert_function = mnist.fill_subparser(subparser)
         args = parser.parse_args(['mnist'])
         args_dict = vars(args)
-        func = args_dict.pop('func')
-        filename, = func(**args_dict)
+        filename, = convert_function(**args_dict)
         h5file = h5py.File(filename, mode='r')
         assert_equal(
             h5file['features'][...],
@@ -167,11 +166,10 @@ class TestMNIST(object):
         subparser = subparsers.add_parser('mnist')
         subparser.set_defaults(
             directory=self.tempdir, output_directory=self.tempdir)
-        mnist.fill_subparser(subparser)
+        convert_function = mnist.fill_subparser(subparser)
         args = parser.parse_args(['mnist'])
         args_dict = vars(args)
-        func = args_dict.pop('func')
-        filename, = func(**args_dict)
+        filename, = convert_function(**args_dict)
         assert_equal(os.path.basename(filename), 'mnist.hdf5')
 
     def test_converter_no_filename(self):
@@ -180,11 +178,10 @@ class TestMNIST(object):
         subparser = subparsers.add_parser('mnist')
         subparser.set_defaults(
             directory=self.tempdir, output_directory=self.tempdir)
-        mnist.fill_subparser(subparser)
+        convert_function = mnist.fill_subparser(subparser)
         args = parser.parse_args(['mnist', '--dtype', 'bool'])
         args_dict = vars(args)
-        func = args_dict.pop('func')
-        filename, = func(**args_dict)
+        filename, = convert_function(**args_dict)
         assert_equal(os.path.basename(filename), 'mnist_bool.hdf5')
 
     def test_wrong_image_magic(self):
@@ -236,11 +233,10 @@ class TestBinarizedMNIST(object):
         subparser.set_defaults(
             directory=self.tempdir, output_directory=self.tempdir,
             output_filename='mock_binarized_mnist.hdf5')
-        binarized_mnist.fill_subparser(subparser)
+        convert_function = binarized_mnist.fill_subparser(subparser)
         args = parser.parse_args(['binarized_mnist'])
         args_dict = vars(args)
-        func = args_dict.pop('func')
-        filename, = func(**args_dict)
+        filename, = convert_function(**args_dict)
         h5file = h5py.File(filename, mode='r')
         assert_equal(h5file['features'][...],
                      numpy.vstack([self.train_mock, self.valid_mock,
@@ -292,11 +288,10 @@ class TestCIFAR10(object):
         subparser.set_defaults(
             directory=self.tempdir, output_directory=self.tempdir,
             output_filename='mock_cifar10.hdf5')
-        cifar10.fill_subparser(subparser)
+        convert_function = cifar10.fill_subparser(subparser)
         args = parser.parse_args(['cifar10'])
         args_dict = vars(args)
-        func = args_dict.pop('func')
-        filename, = func(**args_dict)
+        filename, = convert_function(**args_dict)
         h5file = h5py.File(filename, mode='r')
         assert_equal(
             h5file['features'][...],
@@ -357,11 +352,10 @@ class TestCIFAR100(object):
         subparser.set_defaults(
             directory=self.tempdir, output_directory=self.tempdir,
             output_filename='mock_cifar100.hdf5')
-        cifar100.fill_subparser(subparser)
+        convert_function = cifar100.fill_subparser(subparser)
         args = parser.parse_args(['cifar100'])
         args_dict = vars(args)
-        func = args_dict.pop('func')
-        filename, = func(**args_dict)
+        filename, = convert_function(**args_dict)
         h5file = h5py.File(filename, mode='r')
         assert_equal(
             h5file['features'][...],
@@ -396,9 +390,8 @@ class TestCalTech101Silhouettes(object):
         parser = argparse.ArgumentParser()
         subparsers = parser.add_subparsers()
         subparser = subparsers.add_parser('caltech101_silhouettes')
-        caltech101_silhouettes.fill_subparser(subparser)
-        assert (parser.parse_args(['caltech101_silhouettes', '16']).func is
-                caltech101_silhouettes.convert_silhouettes)
+        convert_function = caltech101_silhouettes.fill_subparser(subparser)
+        assert convert_function is caltech101_silhouettes.convert_silhouettes
 
     def test_download_and_convert(self, size=16):
         tempdir = self.tempdir
@@ -444,8 +437,8 @@ class TestIris(object):
         parser = argparse.ArgumentParser()
         subparsers = parser.add_subparsers()
         subparser = subparsers.add_parser('iris')
-        iris.fill_subparser(subparser)
-        assert parser.parse_args(['iris']).func is iris.convert_iris
+        convert_function = iris.fill_subparser(subparser)
+        assert convert_function is iris.convert_iris
 
     def test_download_and_convert(self):
         tempdir = self.tempdir
@@ -562,14 +555,13 @@ class TestSVHN(object):
         parser = argparse.ArgumentParser()
         subparsers = parser.add_subparsers()
         subparser = subparsers.add_parser('svhn')
-        svhn.fill_subparser(subparser)
+        convert_function = svhn.fill_subparser(subparser)
         subparser.set_defaults(
             directory=self.tempdir, output_directory=self.tempdir,
             output_filename='svhn_format_1.hdf5')
         args = parser.parse_args(['svhn', '1'])
         args_dict = vars(args)
-        func = args_dict.pop('func')
-        filename, = func(**args_dict)
+        filename, = convert_function(**args_dict)
         h5file = h5py.File(filename, mode='r')
 
         expected_features = sum((self.f1_mock[split]['image']
@@ -592,14 +584,13 @@ class TestSVHN(object):
         parser = argparse.ArgumentParser()
         subparsers = parser.add_subparsers()
         subparser = subparsers.add_parser('svhn')
-        svhn.fill_subparser(subparser)
+        convert_function = svhn.fill_subparser(subparser)
         subparser.set_defaults(
             directory=self.tempdir, output_directory=self.tempdir,
             output_filename='svhn_format_2.hdf5')
         args = parser.parse_args(['svhn', '2'])
         args_dict = vars(args)
-        func = args_dict.pop('func')
-        filename, = func(**args_dict)
+        filename, = convert_function(**args_dict)
         h5file = h5py.File(filename, mode='r')
         assert_equal(
             h5file['features'][...],
diff --git a/tests/test_downloaders.py b/tests/test_downloaders.py
index 962d80f..3c12979 100644
--- a/tests/test_downloaders.py
+++ b/tests/test_downloaders.py
@@ -94,20 +94,21 @@ def test_ensure_directory_exists():
 def test_mnist():
     parser = argparse.ArgumentParser()
     subparsers = parser.add_subparsers()
-    mnist.fill_subparser(subparsers.add_parser('mnist'))
+    download_function = mnist.fill_subparser(subparsers.add_parser('mnist'))
     args = parser.parse_args(['mnist'])
     filenames = ['train-images-idx3-ubyte.gz', 'train-labels-idx1-ubyte.gz',
                  't10k-images-idx3-ubyte.gz', 't10k-labels-idx1-ubyte.gz']
     urls = ['http://yann.lecun.com/exdb/mnist/' + f for f in filenames]
     assert_equal(args.filenames, filenames)
     assert_equal(args.urls, urls)
-    assert args.func is default_downloader
+    assert download_function is default_downloader
 
 
 def test_binarized_mnist():
     parser = argparse.ArgumentParser()
     subparsers = parser.add_subparsers()
-    binarized_mnist.fill_subparser(subparsers.add_parser('binarized_mnist'))
+    download_function = binarized_mnist.fill_subparser(
+        subparsers.add_parser('binarized_mnist'))
     args = parser.parse_args(['binarized_mnist'])
     sets = ['train', 'valid', 'test']
     urls = ['http://www.cs.toronto.edu/~larocheh/public/datasets/' +
@@ -115,54 +116,56 @@ def test_binarized_mnist():
     filenames = ['binarized_mnist_{}.amat'.format(s) for s in sets]
     assert_equal(args.filenames, filenames)
     assert_equal(args.urls, urls)
-    assert args.func is default_downloader
+    assert download_function is default_downloader
 
 
 def test_caltech101_silhouettes():
     parser = argparse.ArgumentParser()
     subparsers = parser.add_subparsers()
-    caltech101_silhouettes.fill_subparser(
+    download_function = caltech101_silhouettes.fill_subparser(
         subparsers.add_parser('caltech101_silhouettes'))
     args = parser.parse_args(['caltech101_silhouettes', '16'])
     assert_equal(args.size, 16)
-    assert args.func is caltech101_silhouettes.silhouettes_downloader
+    assert download_function is caltech101_silhouettes.silhouettes_downloader
 
 
 def test_iris():
     parser = argparse.ArgumentParser()
     subparsers = parser.add_subparsers()
-    iris.fill_subparser(subparsers.add_parser('iris'))
+    download_function = iris.fill_subparser(subparsers.add_parser('iris'))
     args = parser.parse_args(['iris'])
     urls = ['https://archive.ics.uci.edu/ml/machine-learning-databases/'
             'iris/iris.data']
     filenames = ['iris.data']
     assert_equal(args.filenames, filenames)
     assert_equal(args.urls, urls)
-    assert args.func is default_downloader
+    assert download_function is default_downloader
 
 
 def test_cifar10():
     parser = argparse.ArgumentParser()
     subparsers = parser.add_subparsers()
-    cifar10.fill_subparser(subparsers.add_parser('cifar10'))
+    download_function = cifar10.fill_subparser(
+        subparsers.add_parser('cifar10'))
     args = parser.parse_args(['cifar10'])
     filenames = ['cifar-10-python.tar.gz']
     urls = ['http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz']
     assert_equal(args.filenames, filenames)
     assert_equal(args.urls, urls)
-    assert args.func is default_downloader
+    assert download_function is default_downloader
 
 
 def test_cifar100():
     parser = argparse.ArgumentParser()
     subparsers = parser.add_subparsers()
-    cifar100.fill_subparser(subparsers.add_parser('cifar100'))
+    download_function = cifar100.fill_subparser(
+        subparsers.add_parser('cifar100'))
     args = parser.parse_args(['cifar100'])
     filenames = ['cifar-100-python.tar.gz']
     urls = ['http://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz']
     assert_equal(args.filenames, filenames)
     assert_equal(args.urls, urls)
-    assert args.func is default_downloader
+    assert download_function is default_downloader
 
 
 class TestSVHNDownloader(object):
@@ -171,19 +174,18 @@ class TestSVHNDownloader(object):
         subparsers = self.parser.add_subparsers()
         subparser = subparsers.add_parser('svhn')
         subparser.set_defaults(directory='./', clear=False)
-        svhn.fill_subparser(subparser)
+        self.download_function = svhn.fill_subparser(subparser)
 
     def test_fill_subparser(self):
         args = self.parser.parse_args(['svhn', '1'])
         assert_equal(args.which_format, 1)
-        assert args.func is svhn.svhn_downloader
+        assert self.download_function is svhn.svhn_downloader
 
     @mock.patch('fuel.downloaders.svhn.default_downloader')
     def test_svhn_downloader_format_1(self, mock_default_downloader):
         args = self.parser.parse_args(['svhn', '1'])
         args_dict = vars(args)
-        func = args_dict.pop('func')
-        func(**args_dict)
+        self.download_function(**args_dict)
         mock_default_downloader.assert_called_with(
             directory='./',
             urls=[None] * 3,
@@ -195,8 +197,7 @@ class TestSVHNDownloader(object):
     def test_svhn_downloader_format_2(self, mock_default_downloader):
         args = self.parser.parse_args(['svhn', '2'])
         args_dict = vars(args)
-        func = args_dict.pop('func')
-        func(**args_dict)
+        self.download_function(**args_dict)
         mock_default_downloader.assert_called_with(
             directory='./',
             urls=[None] * 3,