Optimization removing RemoteCallKey

Fixes #4079