Panel: Make dismissible option work on-the-fly

Fixes gh-7884