Code change:
OK with me.
Display Strings are a superset of Strings, which are a superset of Tokens. This makes APIs more complex, and makes designers have to think about which structure to use. A couple of thoughts: 1) Should we define best practice, that e.g., you should specify Token or String when you specify String -- similar to HTTP? 2) Should we define a new TokenOrString type to encourage this pattern (both in header definitions and APIs)?
Not sure it would be always a best practice, but there are certainly cases where it is. Thus describing the pattern and making it easier to use it sounds right to me.
I'm probably missing something, but why would we want header designers to use TokenOrString? Given, as you say, string is a superset, if you needed to use a string, why allow tokens too when you can just define the header to simply take string?
For brevity. It happens in legacy header fields all the time (dunno whether that's a good reason, though).
This is because legacy fields didn't actually believe token and string are different types, right? Just that strings were optionally unquoted when you don't need quotes. But guidance on the types is for new fields. And, for better or worse, structured fields went with a data model where quoted and unquoted are different types, rather than alternate encodings of the same types. Now that we've made them different, if you're making a new field that needs to represent string values, use strings. It's true this means designers must pick between two similar types. Perhaps making them different was a mistake. (Though making them the same would mean multiple encodings, which is differently bad.) But we've already done that. It's too late to fully merge them because we'll break everyone who was told structures fields was ready to use and treated them differently. Without fully merging them, generic structured fields APIs are stuck using different types for them. So merging them back needs to be a per-field thing, which means fighting with your SF APIs. Instead, the best simplification we can manage is to discourage the TokenOrString half-measure in new fields and say you should just use one or the other.
To the first half of the bug, yes definitely define best practice! Just if the best practice we pick does not align with the data model we picked, we have done something seriously wrong.
We already have best practice about Display Strings in the spec (in that we steer people away from them if they don't need them). We have a separation between Tokens and Strings because they were visible on the wire in pre-existing headers, and we wanted to be able to retrofit / 'pave the cowpaths'. In some cases, they are interchangeable; in others, they are not. I suspect that if we documented best practice for new fields, we'd say "probably use Strings most of the time" because of the underlying reason I raised this issue for -- APIs have very natural mappings for Strings, but for Tokens you probably need some sort of wrapper object or equivalent strategy, so that you can distinguish them from Strings. I'll put together a PR nudging people that way.
Sounds good! So I guess the decision tree is then: Do you need display strings? If yes, use that and only that. Do you have a legacy reason to use tokens? If use, do whatever your legacy reasons need. Otherwise, just use ASCII strings.

Related discussion:
text: Parsers MUST support Tokens with at least 512 characters. <ins> Note that Tokens are defined largely for compatibility with the data model of existing HTTP fields, and may require additional steps to use in some implementations.  As a result, new fields are encouraged to use Strings. </ins> 3.3.5. Byte Sequences can be conveyed in Structured Fields.