CVE: CVE-2024-50095
Published: 2024-11-05
Description: In the Linux kernel, the following vulnerability has been resolved:

RDMA/mad: Improve handling of timed out WRs of mad agent

Current timeout handler of mad agent acquires/releases mad_agent_priv
lock for every timed out WRs. This causes heavy locking contention
when higher no. of WRs are to be handled inside timeout handler.

This leads to softlockup with below trace in some use cases where
rdma-cm path is used to establish connection between peer nodes

Trace:
-----
 BUG: soft lockup - CPU#4 stuck for 26s! [kworker/u128:3:19767]
 CPU: 4 PID: 19767 Comm: kworker/u128:3 Kdump: loaded Tainted: G OE
     -------  ---  5.14.0-427.13.1.el9_4.x86_64 #1
 Hardware name: Dell Inc. PowerEdge R740/01YM03, BIOS 2.4.8 11/26/2019
 Workqueue: ib_mad1 timeout_sends [ib_core]
 RIP: 0010:__do_softirq+0x78/0x2ac
 RSP: 0018:ffffb253449e4f98 EFLAGS: 00000246
 RAX: 00000000ffffffff RBX: [PII_REDACTED] RCX:[PII_REDACTED]00001f
 RDX:[PII_REDACTED]00001d RSI: 000000003d1879ab RDI: fff363b66fd3a86b
 RBP: ffffb253604cbcd8 R08:[PII_REDACTED]635f3b R09: [PII_REDACTED]
 R10: [PII_REDACTED] R11: ffffb253449e4ff8 R12: [PII_REDACTED]
 R13: [PII_REDACTED] R14: [PII_REDACTED] R15: [PII_REDACTED]
 FS:  [PII_REDACTED](0000) GS:ffff8caa1fc80000(0000) knlGS:[PII_REDACTED]
 CS:  0010 DS: 0000 ES: 0000 CR0: [PII_REDACTED]
 CR2: 00007fd9ec9db900 CR3: [PII_REDACTED] CR4:[PII_REDACTED]7706e0
 DR0: [PII_REDACTED] DR1: [PII_REDACTED] DR2: [PII_REDACTED]
 DR3: [PII_REDACTED] DR6: 00000000fffe0ff0 DR7: [PII_REDACTED]
 PKRU: 55555554
 Call Trace:
  <IRQ>
  ? show_trace_log_lvl+0x1c4/0x2df
  ? show_trace_log_lvl+0x1c4/0x2df
  ? __irq_exit_rcu+0xa1/0xc0
  ? watchdog_timer_fn+0x1b2/0x210
  ? __pfx_watchdog_timer_fn+0x10/0x10
  ? __hrtimer_run_queues+0x127/0x2c0
  ? hrtimer_interrupt+0xfc/0x210
  ? __sysvec_apic_timer_interrupt+0x5c/0x110
  ? sysvec_apic_timer_interrupt+0x37/0x90
  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
  ? __do_softirq+0x78/0x2ac
  ? __do_softirq+0x60/0x2ac
  __irq_exit_rcu+0xa1/0xc0
  sysvec_call_function_single+0x72/0x90
  </IRQ>
  <TASK>
  asm_sysvec_call_function_single+0x16/0x20
 RIP: 0010:_raw_spin_unlock_irq+0x14/0x30
 RSP: 0018:ffffb253604cbd88 EFLAGS: 00000247
 RAX:[PII_REDACTED]01960d RBX: [PII_REDACTED] RCX: ffff8cad2a064800
 RDX:[PII_REDACTED]20001b RSI: [PII_REDACTED] RDI: ffff8cad5d39f66c
 RBP: ffff8cad5d39f600 R08: [PII_REDACTED] R09: [PII_REDACTED]
 R10: ffff8caa443e0c00 R11: ffffb253604cbcd8 R12: ffff8cacb8682538
 R13: [PII_REDACTED] R14: ffffb253604cbd90 R15: ffff8cad5d39f66c
  cm_process_send_error+0x122/0x1d0 [ib_cm]
  timeout_sends+0x1dd/0x270 [ib_core]
  process_one_work+0x1e2/0x3b0
  ? __pfx_worker_thread+0x10/0x10
  worker_thread+0x50/0x3a0
  ? __pfx_worker_thread+0x10/0x10
  kthread+0xdd/0x100
  ? __pfx_kthread+0x10/0x10
  ret_from_fork+0x29/0x50
  </TASK>

Simplified timeout handler by creating local list of timed out WRs
and invoke send handler post creating the list. The new method acquires/
releases lock once to fetch the list and hence helps to reduce locking
contetiong when processing higher no. of WRs
CVSS Score: 5.5 (MEDIUM)
References: https://git.kernel.org/stable/c/2a777679b8ccd09a9a65ea0716ef10365179caac; https://git.kernel.org/stable/c/3e799fa463508abe7a738ce5d0f62a8dfd05262a; https://git.kernel.org/stable/c/7022a517bf1ca37ef5a474365bcc5eafd345a13a