SUSE Labs Conference 2019 - How to handle OOM in block layer, setting BLK_STS_RESOURCE or wait ?
Channel: SUSE Labs
Date: 2019-09-18

None

When system is about out of memory,  and memory management code starts to shrink caches from subsystems which registered the shrinker callback, a deadlock may happen for the following scenario,

- Shrinking dirty cache requires to flush the pages onto disk
- Handing I/O in lower layer may allocate memory for request in queue
- The memory allocation for lower layer may require page reclaim in sync way, which will trigger memory management code to invoke shrinker callbacks again.
- No dirty pages can be flushed, then whole thing dead locking.

I encounter this problem because bcache maintains an in-memory cache called mca, when flushing dirty mca pages for OOM situation, the above deadlock can be observed.

The question is, to avoid such deadlock, should block layer (here I mean bcache/dm/md) directly return -ENOMEM to the caller of generci_make_request, or block the caller to wait untill there are enough pages reclaimed and returned to buddy system ?

Returning -ENOMEM can avoid the deadlock, and notify caller that the lower layer is not able to handle I/O due to lack of memory, if the callers checks bi_status value for BLK_STS_RESOURCE they can properly handle such error (e.g, return -ENOMEM to its caller, or wait for a while and re-submit bio again). But most of code only check whether bi_status is 0, if not just returns -EIO, then don't check BLK_STS_RESOURCE.

The other method, blocking the requester and wait for dirty pages to be flushed (when there is still a few available memory) can be compatible with most of legacy code, but for high speed device like bcache, it may introduce unpredictable I/O latency which is unacceptable for low latency workload and the users who care about I/O latency.

So I'd like to discuss with block layer, memory management and file system people, which is the better way to handle OOM in block layer, and look for a way that people may collaborate to solve the problem properly.

Coly Li

https://labsconf.suse.cz/labs2019/talk/YM7SB7