support `%gui qt5` in kernels

same as terminal IPython in #6496
didn't need anything beyond register
support `%matplotlib qt5`

just needed the key, since `%gui qt5` already works
support %matplotlib nbagg

we don't need to do anything, just add keys to a couple of dictionaries
Merge pull request #6587 from minrk/mpl-qt5

support `%matplotlib qt5` and `%matplotlib nbagg`