Unfortunately, I believe this is leaking file descriptors. Imagine:
1) We get a read request for an object and open an fd here.
2) After opening the object, we discover that it is blocked on scrub/backfill/whatever so we put the read op on a wait list.
3) The PG gets moved off of this OSD and the operation is cancelled instead of being completed.

Is there something I'm missing that makes this scenario impossible?