hashfast: Just keep a queue of the 20 most recent work items per core