Clean up JLine threads when no longer needed

This fixes a thread leak in the run task, see #1453.