Postpone ref list lookup to the draw phase

Remove ref_list from the commit struct and look them up when drawing.
This improves the start-up time by ~15% in tests on the Git repository
and also reduces the memory usage a bit.

References #160