Don't let the brush get emptied

Fixes #51