I suppose this is your concurrency control.

The problem is that multiple Yarp workers (processes) will be running on the same machine, typically 3 to 4/dyno, and this doesn't protect against the same resource being downloaded from different workers.

Also, this would probably result in a "thread bomb": you will create lots of threads, possibly faster than they complete since you don't have a safety net.

You could probably get around it by creating a thread pool, bu t having a worker process will probably end up being simpler.