# Histograms are to CDFs as calibration plots are to...

As you know, histograms are decent visualizations for PDFs with lots of samples...

![](https://optimizationprocess.com/calibration-cdf/assets/histo-10k-20bins.png)

10k predictions, 20 bins

...but if there are only a few samples, the histogram-binning choices can matter a lot:

![](https://optimizationprocess.com/calibration-cdf/assets/histo-10-4bins.png)

10 predictions, 4 bins

![](https://optimizationprocess.com/calibration-cdf/assets/histo-10-7bins.png)

same 10 predictions, 7 bins

The binning (a) discards information, and worse, (b) is *mathematically un-aesthetic*.

But a CDF doesn't have this problem!

![](https://optimizationprocess.com/calibration-cdf/assets/cdf-10.png)

same 10 predictions, every data point precisely represented

* * *

If you make a bunch of predictions, and you want to know how well they're calibrated, classically you make a graph like this:

![](https://slatestarcodex.com/blog_images/calibration2019.png)

source: [SSC's 2019 prediction grading](https://slatestarcodex.com/blog_images/calibration2019.png)

But, as with a histogram, this depends on how you bin your predictions.

![](https://optimizationprocess.com/calibration-cdf/assets/calhisto-10bins.png)

100 predictions, 10 bins

![](https://optimizationprocess.com/calibration-cdf/assets/calhisto-30bins.png)

same 100 predictions, 30 bins

Is there some CDF-like equivalent here? Some visualization with no free parameters?

* * *

I asked that question to several people at Arbor Summer Camp. I got three answers:

1.  "You get from a PDF to a CDF by integrating. So, here, analogously, let's integrate (num predictions with confidence < x that came true) minus (expected num predictions with confidence < x that came true)."
2.  (the same thing, said in different words)
3.  (the same thing, said in different words)

If we make a "CDF" for the above 100 predictions by applying these three insights, we get:

![CDF for calibration curves](https://optimizationprocess.com/calibration-cdf/assets/calcdf.png)

[.py](https://optimizationprocess.com/calibration-cdf/assets/calcdf.py)

I find this a little harder to read than the calibration plots above, which I choose to interpret as a good sign, since CDFs are a little harder to read than histograms. The thing to keep in mind, I think, is: when the curve is going up, it's a sign your probabilities are too high; when it's going down, it's a sign your probabilities are too low.

> *Test: how would you describe the problems that this predictor has?*
> 
> ![](https://optimizationprocess.com/calibration-cdf/assets/calcdf-exaggerated-neg.png)
> 
> [*Solution.*](https://itty.bitty.site/#/?eJxtkE1ug0AMha/ylE1aCaX7CNFFolwg6gEGxoCFY5PxUJTbV1AW6c/Ov+97djlWHxopNaYtR9J8RGsJfgsiGJPVoWbhzOQFck/Ync6XHdiR2Fm7Am5r/ccoOO8dwZ07Ze0QEqEjpRREHshm6LnrC8w9C628Jf8PdzpfFlgbRJ5p39IUf2H/csTmQ/k2VuVYlXWqtvBKjWnEfSLPbHrE3IeM2SaJCAr7fPoHxkSRm2xp79v1YjZAeKD3Te/lSrT6Io2wFrcHhHVY/Jnn9cCl6ybTwju8Lntf+bKFyQ==)

(Are there any better visualizations? Maybe. I [looked into this a couple years ago](https://www.lesswrong.com/posts/sa8Qhby63QLCzcCDo/is-there-an-equivalent-of-the-cdf-for-grading-predictions?commentId=eYvAa3xfx2BpAcKDn), but looking back at it, I think this simple "sum(expected-actual predictions with p<x)" graph is at least as compelling as anything I found.)