DotNet Core Threadpool
By: Jai Rathore
Tags: Thread Pool, Scalability, Dotnet, Dotnet Core

If you are reading this chances are that your Dotnet core app is experiencing some performance issues while scaling. And you think thread starvation could be one of the reasons behind it. Let’s take a look into how the Dotnet threadpool works.

Threadpool is the native software level thread management system for Dotnet runtime. It is also the queuing mechanism for all the incoming requests. There is no separate request queuing mechanism in Dotnet core besides the threadpool.

On the hardware level we typically have 1 thread per core of the CPU. So in most cases of our applications and in the context of this post we will consider 16 threads available, assuming we have a 16 core processor. Ideally we want our spawned threads to be at or around the same number. This may be a little confusing but stay with me.

In a fully asynchronous environment, the way Dotnet Threadpool works is :- for an incoming request it spawns a new thread — that thread runs it’s current operation (before async call), saves the current context and adds itself to the global pool as an available thread. When the remaining async call completes or if a new request comes in at this point, the Dotnet threadpool checks if all the 16 threads are already in operation, if yes it spawns up a new thread, if not it takes up one of the available threads from the pool.

Synchronous vs Asynchronous request processing

However if some of the calls in the application are blocking or are not asynchronous, the thread which had run part of the operation can neither add itself to the global pool nor destruct itself. It’s blocked till the entire operation is complete. And if all the current 16 threads are stuck at a given point in a similar fashion, the threadpool will have to spawn a new thread for every incoming request.

If at this point there is a burst of 1000 incoming request , dotnet threadpool will have to spawn a thread for every single one of those request which would result in a lot of threads being spun up. But the CPU can still only process 16 threads at any single point of time. Thread creation, destruction, context switching and their existence itself is very expensive , they eat up a lot of memory and can bring a system to a halt. We do not want so many threads running.

To avoid this Dotnet threadpool has a throttling mechanism. It throttles the incoming request and spawns a new thread at 0.5 second per request. So that means in worst cases if application has a lot of blocking calls and all threads get stuck on the blocking calls, in worst cases when there is a burst of 1000 incoming request a new incoming request/half processed request can wait up to 1000*0.5 = 500 seconds. So in some cases this means that an operation which has to retrieve data from the database, even though the first part of request has retrieved the data from the database — it might have to wait upto 500 seconds to send that data back to the application, which on the application level will show up as a slow query when in reality even though the query execution was pretty fast, it was the thread availability which was the issue. This problem is commonly referred as Thread Starvation.

ThreadPool Default behavior

Thread Pool with Custom Minimum Value

Dotnet threadpool provides 2 settings — MinimumValue and MaximumValue

MaximumValue — total number of threads that can be spawned , which is typically 32,767 (default)

Minimum Value — does not mean the minimum number of threads always present. The application does not boot with the minimum value of threads. The application will still boot with the threads equal to the number of CPU cores. It only means — the minimum number of threads that can be spawned before dotnet start the thread throttling process. We can think of it more in terms of the threshold limit. By default Dotnet sets this value as the number of hardware threads available based on the CPU setting — 16. This means after all 16 threads are in use and 5 new requests come in. Dotnet thread pool will wait 0.5 second for each request then check if a thread becomes free, if yes allots that to the request. If not spawns a new thread. If all the threads are blocked because of some blocking operation — the 5th request with current setting might have to wait 5 * 0.5 = 2.5 seconds to be processed.

If we increase the minimum value to 100. This means that if there is a sudden burst of 100+ request and all the current threads are busy. It will instantly spawn 100 threads for each of those requests and only then start the throttling process. This means there will now be at least 116 threads fighting for the resources on the machine (memory/CPU) which is ideally only designed to handle 16 threads. If the request burst is bigger or if it continues for a longer time, our system can soon become unresponsive and will need a reboot.

We can use minimum value as a patch for sometime, but it is not recommended by Microsoft. However minimum value of 100 should be relatively safe provided our burst of incoming requests is not constant or long lasting. And their recommendation was if the blocking calls cannot be avoided then to use it with some sort of a concurrency limiter which would not take incoming request (return 503) past a certain limit so the system doesn’t become unresponsive.

To Setup custom values for the minimum Value thresholds of Threadpool, we only need to make a small change. For example to set minimum value as 100 for both WorkerThread and IOCP thread, in the ConfigServices of your Startup.cs add this line of code.

ThreadPool.SetMinThreads(100, 100);

The real problem is all the blocking calls (non-async) across our applications. We should clean up on all the instances where we are using blocking calls like .Wait() / .Result() / .GetResult() and try using use await instead.

Use ReadFormAsync instead of .Form()

If running on Dotnet 3.0 + use IAsyncEnumerable<T> instead of IEnumerable<T> if returning from action results( because the latter results in synchronous serialization).

Before setting custom values for the thread we should try to measure the current metrics of the app like the current number of threads in the threadpool. This can be done by using a package like AppMetrics DotnetRuntime. (I will try to cover setting that up in another post). We should play with different values of minimum value of threadpool untill we find that sweep spot. Also look at these recommendations.

However it’s best to leave the threadpool alone and as soon as the expected burst of request is gone, the first step should be to remove all the blocking calls from your apps and to make it all asynchronous as soon as possible.