Fix an ICE for struct patterns that don't refer to a variant or a struct

Fixes #17405.