Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a frequency-based SGD method, where the learning rate is inversely proportional to the token frequency. Theoretical results show the proposed methods outperform SGD in skewed distributions, and empirically verify the effectiveness of the proposed method, and reveal the similarity between Adam/AdaGrad's momentum and frequency.


main_review: The paper investigates the idea of incorporating frequency into SGD for embedding learning, which makes a lof of sense and hasn't been explored before to my knowledge. The theoretical results is interesting in that the proposed method outperforms SGD *WHEN* skewed distribution appeared. And the good thing is that for frequent users/items, the convergence rate remains the same, but much improved for infrequent ones. The empirical results are also very interesting: (i) reveals that Adam/Adagrad mostly implicitly capture the frequency information in momentum; (ii) show great performance improvement over SGD, and very competitive performance against adaptive methods.

I just have a few comments/questions:

(i) it'd be great to have a slice performance analysis, which may reveal that the improvement in infrequent items is even larger.
(ii) for infrequent items, whether a high convergence rate will contribute to overfitting? though the generalization property is beyond the scope of the paper.
(iii) it seems the hidden width of DeepFM is `(16, 16) `, which seems too small to me? Will the analysis/results change for large NNs?



summary_of_the_review: The paper nicely introduces frequency information into SGD, with various interesting theoretical analysis and empirical results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a counter-based learning-rate scheduler for SGD. This algorithm is designed based on the long tail distribution in recommendations and languages. The proposed algorithm enjoys a theoretic guarantee unlike other adaptive learning rate methods, e.g., Adams. Simultaneously, the authors demonstrated that the proposed algorithm achieved comparable empirical performance in two recommendation datasets. Overview, the paper is well-written, which has presented the contributions clearly. 

main_review: Minor concerns: 

1. Why are the lines in Figure 4c identical? 
2. It might be better to zoom in on the y-axis for training loss and AUCs in a smaller region. It is hard to distinguish each method in the current y-axis scale. 
3. How robust is each method towards random seeds and hyper-parameters? Having a variance and confidence interval might help. 
4. Any results for languages? 

summary_of_the_review: This paper is well-written and made solid contributions for adaptive learning-rate in long-tail distribution.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a frequency-aware learning algorithm for embedding learning in recommender systems. The idea of incoporating frequency information into the learning algorithm is very interesting to me. The proposed method is very simple and easy to implement with provable benefits.

main_review: Strengths:
- The paper addresses a very interesting problem about incorporating frequency information into the optimizer for embedding learning in recommender systems. 
- The paper is well-written and the proposed method is well-motivated. 
- The paper demonstrates that the proposed method has provable benefits over SGD.

Weakness: In my opinion, the paper can be improved a lot by re-framing the pitch and enriching the experiments. 
- The title/abs/intro of this paper covers the various areas of embedding learning while the experiments are only on recommendation. I would rather suggest the authors frame the pitch in the area of recommender systems instead of in general embedding learning. 
- To echo with the above points, it will be very interesting if the authors also show some empirical evidence of the proposed method on other embedding learning tasks, like word2vec of knowledge graph embeddings. 
- In Figure 2 (b), it looks like there is a big jump in the red line at the iteration around 1500. Could you explain what was happening there?
- I wonder if you tune the momentum in SGD. 
- Do you have figures of correlation on Movielens-1M? like Figure 3(a) and 3(b)?
- In Figure 4 (d) what's the Y-axis? 
- I don't quite get why "In Figure 4c, 4d we compare the training NE curve CF-SGD and Adagrad, we can see that CF-SGD shows faster convergence than Adagrad during training". Looking at Figure 4c, I only see the same convergence rate of CF-SGD and Adagrad. 
- It would be interesting to see a curve plot of the learning rate for each token v.s. the frequency of the token.

Minor issues (typos, formats): 
- workpiece -> wordpiece?
summary_of_the_review: The paper addresses the problem of adaptive learning rates in recommender systems. The authors propose a novel and simple method to incorporate token frequency information into the learning rate schedule. The authors also show provable benefits over SGD. I think the paper can be further improved by changing the pitch to focus more on recommender systems and enriching the experiments as said in the main reviews.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes two optimization algorithms for recommendation where the token distributions are highly imbalanced. The frequency information is integrated into the optimization algorithms for fast convergence and better performance. The proposed algorithms are easy to understand and implement, and the theoretical analysis for the bounds are provided.
main_review: Strengths
1. The proposed algorithms are easy to implement.
2. Both the online and offline situations are considered.

Weaknesses
The experiments are insufficient. Offline training is also regularly (daily or hourly) required and conducted in industrial recommender systems for capturing long-term interests. It is unknown if FA-SGD outperforms the baselines in offline evaluation. Besides, matching and ranking are two stages in recommender systems. FM and DeepFM are commonly used models in the stage of ranking. It is unknown if CF-SGD shows advantages in the stage of matching. I suggest the authors employ the algorithm to some common matching models such as DSSM for a comprehensive evaluation. In addition, only the cross-entropy (point-wise) loss is tested in the experiments. It is unknown if the proposed algorithms can generalize to adapt to other losses such as the pair-wise loss.

summary_of_the_review: This paper proposes to integrate the frequency information of tokens into the optimization algorithms for a fast convergence. The idea is novel and the proposed algorithms are easy-to-implement. I recommend to accept it. However, the experiments are not comprehensive and I want to see a thorough evaluation which considers the offline training and the matching stage. Besides, more loss functions should also be investigated.
review_title: null

Meta-review: The paper provides a new learning technique for problems that require learning embeddings. In particular, the authors analyze a technique that takes into account the frequency of items in an embedding layer to modify the learning rate for each embedding. The paper provides a theoretical analysis of this approached and contrasts it to that of SGD. It also provides experiments validating this approach empirically.

The reviewers agree that the paper provides a simple yet effective method, based on realistic assumptions (non-uniform frequencies). In addition, the paper seems to be well written and easy to follow.
One issue raised in the reviews was about the focus of the paper, and the fact that the experiments are limited to recommendation systems even though the method is claimed to be generic for any model requiring embeddings. During the rebuttal the authors provided experiments for an NLP task that show favorable results to the new technique in another regime. Given the overall positive feedback and this new evidence validating the proposed method, I recommend accepting the paper.
Decision: Accept (Poster)