Track the top N (or bottom N) values added, and ignore the rest