Only user UserSimilarity when necassary