Given the difficulties with exception handling and (not) releasing the semaphore, this might not be worth doing now.  The threads won't really hang around forever unless the active requests take forever, in which case we've got bigger problems.