Implementing Dataset Sources 2021-03-04
Channel: DFFML
Date: 05/03/2021

Yash and John met and worked out how existing datasets could be integrated into DFFML as sources.

We came up with a way to leverage the existing source infrastructure. We wrote new WrapperSource(Context) which can be used to wrap an existing instantiated source.

We also came up with a decorator which allows us to wrap functions (context managers) and create sources out of them.

We implemented a source for the iris training dataset.

Design decisions behind existing source infrastructure were touched on, as well as config infrastructure and the importance of unification there. Entrypoints and plugins in general were also touched on.