Remove cross-borrowing for traits

Closes #15349

r? @pcwalton (or anyone else)