Handle custom deref returning fat pointers

Closes #16930 

r?