Clean up worker command and response queues