Code change:
Servers MUST NOT send a REGISTER_DATAGRAM_CONTEXT capsule on a stream before they have received at least one REGISTER_DATAGRAM_CONTEXT <del> capsule from the client on that stream.  This ensures that clients control whether datagrams are allowed for a given request.  If a </del> <ins> capsule or one REGISTER_DATAGRAM_NO_CONTEXT capsule from the client on that stream.  This ensures that clients control whether datagrams are allowed for a given request.  If a client receives a REGISTER_DATAGRAM_CONTEXT capsule on a stream where the client has not yet sent a REGISTER_DATAGRAM_CONTEXT capsule, the client MUST abruptly terminate the corresponding stream with a stream error of type H3_GENERAL_PROTOCOL_ERROR. Servers MUST NOT send a REGISTER_DATAGRAM_CONTEXT capsule on a stream where it has received a REGISTER_DATAGRAM_NO_CONTEXT capsule.  If a </ins> client receives a REGISTER_DATAGRAM_CONTEXT capsule on a stream where <del> the client has not yet sent a REGISTER_DATAGRAM_CONTEXT capsule, the </del> <ins> the client has sent a REGISTER_DATAGRAM_NO_CONTEXT capsule, the </ins> client MUST abruptly terminate the corresponding stream with a stream error of type H3_GENERAL_PROTOCOL_ERROR. 4.2. <ins> The REGISTER_DATAGRAM_NO_CONTEXT capsule (type=0x03) allows a client to inform the server that datagram contexts will not be used with this stream.  It also informs the server of the encoding and semantics of datagrams associated with this stream.  Its Capsule Data field consists of: A string of comma-separated key-value pairs to enable extensibility, see the definition of the same field in register- capsule for details. Note that this registration is unilateral and bidirectional: the client unilaterally defines the semantics it will apply to the datagrams it sends and receives with this stream. Endpoints MUST NOT send DATAGRAM frames without a Context ID until they have either sent or received a REGISTER_DATAGRAM_NO_CONTEXT Capsule.  However, due to reordering, an endpoint that receives a DATAGRAM frame before receiving either a REGISTER_DATAGRAM_CONTEXT capsule or a REGISTER_DATAGRAM_NO_CONTEXT capsule MUST NOT treat it as an error, it SHALL instead drop the DATAGRAM frame silently, or buffer it temporarily while awaiting a REGISTER_DATAGRAM_NO_CONTEXT capsule or the corresponding REGISTER_DATAGRAM_CONTEXT capsule. Servers MUST NOT send the REGISTER_DATAGRAM_NO_CONTEXT capsule.  If a client receives a REGISTER_DATAGRAM_NO_CONTEXT capsule, the client MUST abruptly terminate the corresponding stream with a stream error of type H3_GENERAL_PROTOCOL_ERROR. Clients MUST NOT send more than one REGISTER_DATAGRAM_NO_CONTEXT capsule on a stream.  If a server receives a second REGISTER_DATAGRAM_NO_CONTEXT capsule on the same stream, the server MUST abruptly terminate the corresponding stream with a stream error of type H3_GENERAL_PROTOCOL_ERROR. Clients MUST NOT send a REGISTER_DATAGRAM_NO_CONTEXT capsule on a stream before they have sent at least one HEADERS frame on that stream.  This removes the need to buffer REGISTER_DATAGRAM_CONTEXT capsules when the server needs information from headers to determine how to react to the capsule.  If a server receives a REGISTER_DATAGRAM_NO_CONTEXT capsule on a stream that hasn't yet received a HEADERS frame, the server MUST abruptly terminate the corresponding stream with a stream error of type H3_GENERAL_PROTOCOL_ERROR. Clients MUST NOT send both REGISTER_DATAGRAM_CONTEXT capsules and REGISTER_DATAGRAM_NO_CONTEXT capsules on the same stream.  If a server receives both a REGISTER_DATAGRAM_CONTEXT capsule and a REGISTER_DATAGRAM_NO_CONTEXT capsule on the same stream, the server MUST abruptly terminate the corresponding stream with a stream error of type H3_GENERAL_PROTOCOL_ERROR. 4.3. </ins> The CLOSE_DATAGRAM_CONTEXT capsule (type=0x01) allows an endpoint to inform its peer that it will no longer send or parse received datagrams associated with a given context ID.  Its Capsule Data field

Related discussion:
text: I think the second option "Negotiate the presence of context IDs using an HTTP header" is the best personally. In terms of: "Context IDs would still be mandatory to implement on servers because the client might send that header", I assume server in this context is the final application server, not a proxy? Even so, I'd argue it's applications which decide if they need this functionality, so if a server only implemented one application and it didn't use Context-IDs, then I don't see why it would need to implement it.
Ian, in your comment, are you talking about an intermediary when you say "proxy"? If so, then an intermediary need not implement anything based on the payload of than H3 DATAGRAM, it only has to pass the data along. That may be worth a clarification.
Having context ID optional has a nice future-proof property - other uses of datagrams in HTTP/3 can decide if they want the second layer or not. I wouldn't even state it as application needing multiplexing or not. Other applications might find it very useful to include a second ((third, fourth, fifth) layer of frame header. I think its fine for us to state as much in this in the spec, while also defining an explicit second layer for multiplexing.
Yes, by proxy I meant intermediary. And I agree about what intermediaries need to do, I just wanted to ensure I understood what server meant when David used it above. Yup, whenever all this gets written up, we can make it clear that intermediaries can ignore much of this.
The method defines the semantics of datagrams for a particular request, thus I think the third option makes most sense. I don't see a point in using explicit negotiation, since in the two cases I can think of the top of the head, the answer is always clear from the method (CONNECT-IP seems to always want context IDs, WebTransport does not ever want context IDs). If some method starts out without context, but wants to add context later, it can add its own negotiation.
One set of use cases for which the method approach(3) is best are those where there are multiple context IDs but they're predefined by the method and don't need negotiation. My strawman is WebRTC.
Looks good overall.