Improve performance of TopicModel::ApplyDiff