Code change:
This responds to an AD question in review. The text is proposed to shed some light on how to go about evaluating a new CC algorithm, and when to use different approaches. It could be enough, or it could be improved, please think about how best to say this.
Thanks for this.. this looks like very good expression of the expectations on how one can think of evaluating the candidate congestion control algorithms without going into lengthy details. For the sake of clarity, it seems like the word "overload" (in the quoted sentence below) needs some explanation on what exactly we mean here - I am assuming we are referring to typical congestion scenarios on the path due to packet inflow and out flow mismatch in a bottleneck network node. >>"For many algorithms, an initial evaluation will consider individual protocol mechanisms in a simulator to analyse their stability and safety across a wide range of conditions, including overload".
overload, aka a lot more traffic than the path can handle... rather than typical congestion. Martin may have suggested wording...
While section 5 expects to do well-round of evaluations, it does not specify where or how to do the evaluation. Does the congestion control proposals provide the evaluation results in a simulator or in a real lab setup or in a controlled part of the Internet or over the Internet? While it is understandable that an accurate description on the evaluation environment is tough to define but the specification should at least mention what is good enough environment to evaluation on, or state it does not really matter.
See below:On 27 May 2024, at 10:50, Zaheduzzaman Sarker NAME wrote: While section 5 expects to do well-round of evaluations, it does not specify where or how to do the evaluation. Does the congestion control proposals provide the evaluation results in a simulator or in a real lab setup or in a controlled part of the Internet or over the Internet? While it is understandable that an accurate description on the evaluation environment is tough to define but the specification should at least mention what is good enough environment to evaluation on, or state it does not really matter. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: NAME I am not sure how we add guidance … surely all of these method can be appropriate (more so for some of the evaluations) when bringing new work to the IETF. When last calling a draft - it would be good to see more than one approach where possible. Simulation ought to be better in understanding stability and identifying worst case impact; real lab or controlled tested could be good for understanding typical interactions with other traffic. I still wonder what we can best advise?Gorry Postscript: Reading again: my comment is about testing specific new CC mechanisms, if we're talking about a new CC algorithm (not just a new technique), then I agree with Martin below.
This is covered in Section 3, which is becoming Section 2: Is that good enough?
I have noticed that section. I think it is great that we are somewhat defining the criteria for a PS. But still we are not giving any hint on - say, if simulaiton results are good enough to be considered as experimental or shall we at least want to see some controlled testbed evaluations, or put it like this - what is considered as bare minimum empirical evidence? I would be great if we can put some clarity here.
I think we ought to expect some experiment results ... but how detailed, rather depends on what is being proposed. Having some experiments (even at large scale) and no simulation of theory (e.g. on how a method treats overload) would also seem very dangerous to me. These have always been the case in the past - i.e. TSVWG has pushed back to ICCRG to see experimental results AND some simulation/analysis when it comes to CC.
NAME are you saying we are just fine with some expection on evidence of experiments without specifying what is the minimum required environ to test in? we do have test cases defined for new algorithm testing, can't we at least say get inspiration or use them in the experiments?
Aha - I think you're suggesting we provide examples (rather than requirements) and we could refer to RFCs? If so, I'll try a PR.
I am not 100% sure this is necessary but given the AD review I'm happy to include it now.Just one little nit, but it looks good.

Related discussion:
text: 3. <ins> 3.1. This document does not specify specific evaluation methods, short of internet-scale deployment and measurement, to test the criteria described below.  There are multiple possible approaches to evaluation.  Each has a role, and the most appropriate approach depends on the criteria being evaluated and the maturity of the specification. For many algorithms, an initial evaluation will consider individual protocol mechanisms in a simulator to analyse their stability and safety across a wide range of conditions, including overload.  For example, RFC8869 describes evaluation test cases for interactive real-time media over wireless networks.  Such results could also be published or discussed in IRTF research groups, such as ICCRG and MAPRG. Before a proposed congestion control algorithm is published as an Experimental or Standards Track RFC, the community SHOULD gain practical experience with implementation and experience using the algorithm.  Where there is implementation by independent teams, this can help provide assurance that a specification has avoided assumptions or ambiguity.  An independent evaluation by multiple teams helps provide assurance that the design meets the evaluation criteria, and can assess typical interactions with other traffic. This evaluation could use an emulated laboratory environment or a controlled experiment (within a limited domain or at Internet-scale). Evidence of results is normally considered by the working group in deciding if a specification is ready for publication and ought to be documented in any request for the working group to publish the specification. 3.2. </ins> This document applies to proposals for congestion control algorithms that seek Experimental or Standards Track status.  Evaluation of both cases involves the same questions, but with different expectations