Improve percentiles computation with HDR histogram

This commit also introduces new fields in the Metrics.Latencies struct.
* Min
* P999

> On 11 Nov 2014, at 15:10, Gil Tene <notifications@github.com> wrote:
> 
> In lib/metrics.go:
> 
> >  		}
> >  		if result.Error != "" {
> >  			errorSet[result.Error] = struct{}{}
> >  		}
> >  	}
> >  
> > +	hist := hdr.New(int64(m.Latencies.Min), int64(m.Latencies.Max), 5)
> > +	for _, result := range r {
> > +		hist.RecordValue(int64(result.Latency))
> The RecordCorrectedValue math/logic will work if you know that there were no interleaved requests that happened during the expected interval (e.g. it is valid within each serial request issuing connection), and therefore know that there are a specific number of "missing" requests that should have been sent but weren't. But it's not clear to me that you attacker knows that (can there be another attacker sending on the same connection?).
> 
> 
Vegeta differs from HTTP load testing tools in the sense that it maintains a user specified request rate for a given duration, instead of issuing as many requests as fast as possible. It answers “how do my latencies look like at this request rate” instead of the usual “how many can I do?” which is benchmarking.
I am not sure this is what you meant by “connection, but an attack is executed by a single HTTP client, by all go-routines. This HTTP client can optionally disable HTTP Keepalive which will force a new TCP connection on every request/hit.
> Given that your goal is to maintain a constant throughout, I would avoid relying on the scheduler to do time math for you, or on compensation inside the histogram recording. I would instead modify the code to make decisions based on actual time accounting, and to make sure that the right number of requests is actually sent, timing the latency of each from the time it was supposed to be sent until it actually responded. The simplest way to do that is to keep issuing requests as long as "not enough requests have been issued up to now". In order to avoid back-to-back sends when "behind" (which can create an unrealistic huge spike after a pause on either the sender or receiver), it is usually useful to do "catchup" at a limited throughout (e.g. 2x the normal throughput).
> 
I am not relying on the scheduler to do time math. I am relying on a high precision timer which ticks at the required request rate. There are pre-scheduled go-routines which instantly issue a request on each tick, regardless of whether previous requests have returned. I’d say the only way to fall behind is to have a massive GC pause, in which case your algorithm would be immensely helpful. Until today, I haven’t seen a single tick dropped, under various heavy load generation conditions. But I suppose it is possible. I have to investigate further.
> This usually involves keeping track of how many requests have actually been completed in a "connection", and doing the math based purely on that count, the required throughput (in that connection), and the current time.
> 
> E.g. can have the hit daisy-chain within a "connection" by kicking off another hit each time they are done doing what they need to (scheduled for when the next request in this "connection" needs to be send according to the connection's required throughput) . What a hit "needs to do" is to either send the request (if the time has arrived) or skip the request if it is not yet time to send it. The hit should always re-schedule another hit for the proper time to send the next request (which may be "now").
> 
> You can see an example of logic to calculate the time until the next request should be sent here: https://github.com/giltene/wrk2/blob/master/src/wrk.c#L461 <https://github.com/giltene/wrk2/blob/master/src/wrk.c#L461>
> It basically uses the current number of completed requests and the current time to decide how long to wait before the next request sis sent (or 0 if it is time to send the request). Providing the time until the next send rather than just a "send now" boolean indication is useful because it allows you to sleep/wait/delay checking again until that time arrives.
> 
> You can see logic for computing the latency when a response is complete her:
> https://github.com/giltene/wrk2/blob/master/src/wrk.c#L539 <https://github.com/giltene/wrk2/blob/master/src/wrk.c#L539>
> (ignore all the nice debug-related output and the u_latency_histogram math, focus on the calculation that feeds the recoding into latency_histogram).
> 
> If you use this logic, your throughput (averaged over time) will remain constant, catching up to missed requests at an increased (but not back to back) rate. With a such a constant throughout plan actually being executed, your latency reporting will be precise, since you have a known plan (to execute requests at a given constant pace) and can determine exactly what the latency is for each response based purely on it's arrival time (ignoring its actual send time, which is susceptible to coordination when you are temporarily off-pace).
> 
> 

Thank you so much for the detailed insight. I really appreciate it. Before I dive into this though, I’d like to first simulate a load test where the potential issues with my current approach become obvious. If you have anything in mind for this, please let me know!
> —
> Reply to this email directly or view it on GitHub <https://github.com/tsenart/vegeta/pull/92/files#r20151165>.
>