Ignore RPC calls from components that are concurrently removed (#12337)

Change-Id: I8b97444d33b9535b9073fd705fed15a6cc2992e7