SlowGuess/ACL_2020_OCR huggingface dataset:
First, the methods based on average pooling are usually inferior to those using other pooling methods in our experiments. This is probably because average pooling equally regards different features and cannot distinguish their informativeness. Thus, modeling the importance of different features has the potential to improve text representation learning. Second, the methods based on attentive pooling outperform their variants based on max pooling. This may be because attentive pooling can model the informativeness of contexts for text representation, while max pooling only selects the most salient features, which may be sub-optimal. Third, our APLN approach can consistently outperform other pooling methods, and further hypothesis test results show that the improvement brought by our approach is significant  $(p < 0.01)$ . This may be because vanilla max pooling and attentive pooling methods use a fixed pooling norm for universal text representation learning, and the differences in the characteristics of different tasks and datasets are not considered, which may also be sub-optimal. Our approach can dynamically adapt the pooling norm in different scenarios, which may facilitate text representation learning. In addition, we find the advantage in Macro-F score of our approach over other methods is more significant on the datasets with imbalanced class distributions. This may be because our approach can build text representation in a softer manner, which may help neural models avoid focusing on the clues of major classes only and alleviate their dominance. Fourth, we find hierarchical models (HAN and HAN-APLN) outperform flatten models (e.g., LSTM-APLN) for doc
Methods	IMDB	Amazon	Yelp
CNN-Avg	1.388	0.920	0.847
CNN-Max	1.322	0.908	0.834
CNN-Att	1.292	0.899	0.824
CNN-APLN	1.271	0.886	0.801
LSTM-Last	1.316	0.896	0.822
LSTM-Avg	1.343	0.911	0.830
LSTM-Max	1.269	0.890	0.815
LSTM-Att	1.257	0.878	0.799
LSTM-APLN	1.233	0.865	0.784
HAN	1.230	0.866	0.789
HAN-APLN	1.214	0.858	0.776
ument representation learning. This may be because modeling documents in a hierarchical manner can better utilize the structure of documents. In addition, since our approach can be applied at both word and sentence levels in HAN, text representation may be learned more accurately. These results validate the effectiveness of our approach.
To further validate the generality of our approach in regression tasks $^{8}$ , we also conduct experiments on the IMDB, Amazon and Yelp datasets by formulating the task as a rating regression problem, and the results in terms of RMSE are shown in Table 3. From the results, we find our APLN approach can also bring consistent improvements to many existing methods in the regression task.
5.3 Influence of Scale Limiting and Re-formulation
In this section, we will explore the influence of the scale limiting and re-formulation techniques on the stability and relative pooling speed of our approach. The results are summarized in Table 4. From these results, if the limitation of nonnegativity is removed, the model training is usually unstable, which is intuitive. In addition, if the scale limitation  $(\leq 1)$  is removed, our model occasionally does not converge. This may be because when  $p > 1$ , our model has the risk of gradient explosion. Thus, the scale of input features should be limited. Besides, the re-formulation method also has critical impacts on our approach. This is probably because directly computing the real-valued
exponents of input features may be numerically unstable. In our approach we decompose the exponents into three stable operations, which is robust to numerical errors. In addition, the pooling speed can be effectively improved, since the computational costs of these atomic operations are usually small. These results validate the effectiveness of our approach.
Table 3: The performance of different methods on rating regression. Lower RMSE scores indicate better performance.
Stability	Speed
-SL (≥ 0)	×	1.001
-SL (≤ 1)	○	1.001
-RF	×	0.116
APLN	✓	1.000
Table 4: Influence of the scale limiting (abbreviated as  $SL$ ) and re-formulation (abbreviated as  $RF$ ) on the stability and relative pooling speed of APLN. The symbol  $\circ$  represents the model training is unstable on occasion.
5.4 Influence of Norm Initialization
In this section, we study the influence of a small but very important step, i.e., the initialization of the trainable pooling norm  $p$ , on the performance of our approach. We compare the performance of LSTM-APLN by varying the initialized values of  $p$ . The results are shown in Fig. 6. From Fig. 6, we find the performance of our approach increases when the initialized value of  $p$  increases. This is intuitive because when  $p$  is too small, the attention network may not be capable of recognizing important contexts effectively, which is not optimal for learning accurate text representations. In addition, when  $p$  is initialized with a too large value, the performance will start to decline. This is probably because a large value of  $p$  will lead to sharp attentions on critical contexts, and other useful information is not fully exploited. Thus, the performance is also not optimal. These results show that a moderate value (e.g., 1.0) is the most appropriate for initializing the pooling norm  $p$ , which is also consistent with standard attentive pooling.
5.5 Parameter Analysis
In this section, we analyze a critical parameter learned by our model, i.e., the pooling norm  $p$  in the APLN module. The evolution of the values of  $p$  learned by LSTM-APLN on the four benchmark datasets during model training is portrayed in Fig. 7. From the results, we have several interesting observations. First, the pooling norms learned by our model are consistently less than
<sup>8</sup>We find that the regression labels need to be normalized, or the performance may be sub-optimal.
Figure 6: The influence of the initialization of the pooling norm  $p$  on our approach.
Figure 7: The evolution of the pooling norm  $p$  learned by our model on different datasets. The required training epochs to achieve the best performance are marked as the grey region.
1, which indicates that our norm-wise attention is "softer" than vanilla attention. This may be because L1 norm is not optimal for attentive pooling, and a softer attention manner may be more suitable for learning accurate text representations. Second, we find it is interesting that the norm  $p$  consistently decreases when the training epoch increases. This may be because the model may tend to take the global contexts into consideration rather than focus on important ones. Third, a moderate norm  $p$  is more appropriate for our approach. This may be because when  $p$  is too large, the attentions may be too sparse and useful contextual information is not fully exploited. When  $p$  is too small, the attention networks cannot effectively distinguish informative contexts from uninformative ones, which may also be sub-optimal for learning text representations. Fourth, we observe that the norm  $p$  learned on datasets with imbalanced class distributions is lower than those with balanced distributions. This may be because on imbalanced dataset, if  $p$  is too large, the clues
(a) Attention weights in HAN. Predicted rating is 4.
(b) Attention weights in HAN-APLN. Predicted rating is 3.
Figure 8: Visualization of the word-level and sentence-level attention weights in HAN and HAN-APLN on a randomly selected review in the Amazon dataset, whose gold rating score is 3. Darker colors indicate higher attention weights. The visualized attention weights of APLN are  $\alpha_{i}^{p}$  of words and sentences, under  $p = 0.885$  at the word level and  $p = 0.892$  at the sentence level.
of the majority classes may be over-emphasized, and other useful information is not fully respected. Thus, the performance of our APLN approach is better when it learns a moderate pooling norm.
In this section, we conducted several case studies to further explore the effectiveness of our APLN approach. We visualize the word-level and sentence-level attention weights in HAN and HAN-APLN of a randomly selected review to compare their differences, and the results are portrayed in Fig. 8. According to the results, we have several observations. First, both HAN and HAN-APLN can recognize important words and sentences. For example, the word "liked" and the sentence "I really liked the case, at first." are highlighted since they are important for modeling the opinions condensed by this review. Second, the attentions of HAN are sparse, which indicates that HAN tends to focus more on some contexts in a review such as the first and the third sentence, and pays little attentions to the useful information in other contexts such as the fourth and fifth sentences. In addition, HAN wrongly classifies the rating of this review. This is probably because the rating of a review is usually a synthesis of all opinions conveyed by it. Thus, it may not be optimal for learning accurate text representations if only salient contexts are considered. Third, different from HAN, the attentions of HAN-APLN are smoother. This is probably because the pooling norm learned by our approach is less than 1, which encourages our model
to attend to important contexts in a softer manner. In addition, HAN-APLN can classify this review correctly. This is probably because our approach can effectively take global contextual information into consideration, and does not over-emphasize critical contexts. Thus, our APLN approach can learn more accurate text representations than the methods based on vanilla attentive pooling. These results show the effectiveness of our approach.
In this paper, we propose an Attentive Pooling with Learnable Norms (APLN) approach for text representation. Instead of using a fixed pooling norm for universal text representation learning, we propose to learn the norm in an end-to-end framework to automatically find the optimal ones for learning text representations in different tasks. In addition, we propose two methods to ensure the numerical stability of the model training. The first one is scale limiting, which limits the scale of input representations to ensure their non-negativity and avoid potential exponential explosion. The second one is re-formulation, which decomposes the exponent operation into several safe atomic operations to avoid computing the real-valued powers of input features with less computational cost. Extensive experiments on four benchmark datasets validate the effectiveness of our approach.
In our future work, we will explore several potential directions. First, we plan to explore why the model prefers "soft" attentions rather than "hard" ones, which is different from the findings in several prior works based on hard attention. Second, we plan to study how to model the differences on the characteristics of different samples and use different pooling norms, which may have the potential to further improve our approach. Third, we will explore how to generalize our approach to other modalities, such as images, audios and videos, to see whether it can facilitate more attention-based methods.
This work was supported by the National Key Research and Development Program of China under Grant number 2018YFC1604002, the National Natural Science Foundation of China under Grant numbers U1936208, U1936216, U1836204, and U1705261.
Huimin Chen, Maosong Sun, Cunchao Tu, Yankai Lin, and Zhiyuan Liu. 2016. Neural sentiment classification with user and product attention. In EMNLP, pages 1650-1659.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of machine learning research, 12(Aug):2493-2537.
Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. 2017. Attention-over-attention neural networks for reading comprehension. In ACL, pages 593-602.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*, pages 4171–4186.
Qiming Diao, Minghui Qiu, Chao-Yuan Wu, Alexander J Smola, Jing Jiang, and Chong Wang. 2014. Jointly modeling aspects, ratings and sentiments for movie recommendation (jmars). In KDD, pages 193-202. ACM.
Jiachen Du, Lin Gui, Ruifeng Xu, and Yulan He. 2017. A convolutional attention model for text classification. In NLPCC, pages 183-195. Springer.
Yuyun Gong and Qi Zhang. 2016. Hashtag recommendation using attention-based convolutional neural network. In *IJCAI*, pages 2782-2788.
Caglar Gulcehre, Kyunghyun Cho, Razvan Pascanu, and Yoshua Bengio. 2014. Learned-norm pooling for deep feedforward and recurrent neural networks. In ECML-PKDD, pages 530-546. Springer.
Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In WWW, pages 507-517.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012. Improving word representations via global context and multiple word prototypes. In ACL, pages 873-882.
Rie Johnson and Tong Zhang. 2015. Semi-supervised convolutional neural networks for text categorization via region embedding. In NIPS, pages 919-927.
Rie Johnson and Tong Zhang. 2016. Supervised and semi-supervised text categorization using LSTM for region embeddings. In ICML, pages 526-534.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP, pages 1746-1751.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Recurrent convolutional neural networks for text classification. In AAAI.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature, 521(7553):436.
Zheng Li, Ying Wei, Yu Zhang, and Qiang Yang. 2018. Hierarchical attention transfer network for cross-domain sentiment classification. In AAAI.
Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2017. Interactive attention networks for aspect-level sentiment classification. In AAAI, pages 4068-4074.
Hao Peng, Jianxin Li, Yu He, Yaopeng Liu, Mengjiao Bao, Lihong Wang, Yangqiu Song, and Qiang Yang. 2018. Large-scale hierarchical text classification with recursively regularized deep graph-cnn. In WWW, pages 1063-1072.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL-HLT, pages 2227-2237.
Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 15(1):1929-1958.
Duyu Tang, Bing Qin, and Ting Liu. 2015a. Document modeling with gated recurrent neural network for sentiment classification. In EMNLP, pages 1422-1432.
Duyu Tang, Bing Qin, and Ting Liu. 2015b. Learning semantic representations of users and products for document level sentiment classification. In ACLIJCNLP, pages 1014-1023.
Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014. Learning sentiment-specific word embedding for twitter sentiment classification. In ACL, pages 1555-1565.
Hanqing Tao, Shiwei Tong, Hongke Zhao, Tong Xu, Binbin Jin, and Qi Liu. 2019. A radical-aware attention-based model for chinese text classification. In AAAI.
Chuhan Wu, Fangzhao Wu, Junxin Liu, Shaojian He, Yongfeng Huang, and Xing Xie. 2019a. Neural demographic prediction using search query. In WSDM, pages 654-662.
Chuhan Wu, Fangzhao Wu, Junxin Liu, and Yongfeng Huang. 2019b. Hierarchical user and item representation with three-tier attention for recommendation. In NAACL-HLT, pages 1818-1826.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In NAACL-HLT, pages 1480-1489.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NIPS, pages 649-657.
Xue Zhao, Chao Wang, Zhifan Yang, Ying Zhang, and Xiaojie Yuan. 2016. Online news emotion prediction with bidirectional LSTM. In International Conference on Web-Age Information Management, pages 238-250. Springer.
Xinjie Zhou, Xiaojun Wan, and Jianguo Xiao. 2016. Attention-based LSTM network for cross-lingual sentiment classification. In EMNLP, pages 247-256.