Handle short ranges with few granularity results

Fixes a bug when we assumed there were always results for years,
months, days, etc.