fix nbconvert filter validation
use register_filter() for registering user filters
Merge pull request #3512 from jakevdp/nbconvert_filter

fix nbconvert filter validation