Use proper datastructure to make size() calls fast again