Terminate executors that aren't needed.

If we launch an executor and then kill the task immediately after, make
sure we also terminate the executor when there are no other tasks.

Review: https://reviews.apache.org/r/11125