Stop using all_related

While this is faster with smaller amounts of data, it slows down with
huge tables.