Hide instead of removing config options

Fixes #532