Remove optional dependencies

Some work in progress for #115. Please do not merge for now.