# Benchmarking Deep Learning Methods for Aspect Level Sentiment Classification
## 4 Experimental Setup and Datasets
### Characteristics of Datasets  
In this study, the experimental evaluation is carried out on 8 benchmark datasets of different domains that are Restuaurnt14, Laptop14, Restaurant15, Restaurant16, Twitter, Sentihood, Mitchell, and MAMS. All the datasets except Sentihood are of 3-way polarity. The 3-way polarity means, for these datasets, each aspect term can belong to the positive, negative, or neutral category. Table 2. shows the statistics regarding the number of positive, negative, and neutral samples in each dataset (train and test separately).  
In the literature of ALSC, the majority of the proposed deep learning methods are evaluated on the datasets released by the International Workshop for Semantic Evaluation (SemEval). Restaurant 14 and Laptop14 datasets released in SemEval 2014 [46] task are the most popular datasets. In continuation of previous workshops, two more datasets of the restaurant domain are released by SemEval 2015 [47] and SemEval 2016 [13] named as Restaurant 15 and Restaurant 16. Another popular dataset in ALSC literature is Twitter [48]. It is the data derived from tweets specifically and is also known as target-dependent sentiment classification data.  
The other three datasets included in this study are Mitchell [49], Sentihood [50], and MAMS [31] dataset from Twitter, neighborhood, and restaurant domain, respectively. Mitchell dataset is the tweets data originally released for the English and Spanish languages. In this study, the English language sentences of the dataset are evaluated. Sentihood data is obtained from the yahoo platform. The data is related to the aspects discussed in the neighborhood of London city. This dataset is of two-way polarity.  
MAMS(Multi-Aspect-Multi-Sentiment) is the latest dataset in ALSC literature. MAMS dataset is obtained from the CitySearch New York dataset [51] by manually annotating the aspect terms in the sentences with their polarity. This dataset can be called a challenging dataset because each sentence in the dataset is having multiple aspects with different polarities. Thus, handling of context-aspect relationship for any method will be more critical. Although, the other datasets in ALSC literature also have such multi-aspect sentences but the number of such sentences is quite low. In addition to this, the MAMS  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & \multirow{2}{*}{**Positive Samples**} & \multicolumn{2}{c}{**Negative**} & \multicolumn{2}{c}{**Neutral**} \\  & & & \multicolumn{2}{c}{**Samples**} & \multicolumn{2}{c}{**Samples**} \\ \hline  & **Train** & **Test** & **Train** & **Test** & **Train** & **Test** \\ \hline Rest14 & 2164 & 728 & 805 & 196 & 633 & 196 \\ \hline Lap14 & 987 & 341 & 866 & 128 & 460 & 169 \\ \hline Twitter & 1561 & 173 & 1560 & 173 & 3127 & 3146 \\ \hline Rest15 & 1198 & 454 & 403 & 346 & 53 & 45 \\ \hline Rest16 & 1657 & 611 & 749 & 204 & 101 & 44 \\ \hline Sentihood & 2480 & 1217 & 921 & 462 & - & - \\ \hline Mitchell & 695 & 695 & 269 & 269 & 2259 & 2259 \\ \hline MAMS & 3380 & 400 & 2764 & 329 & 5042 & 607 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Characteristics of the Datasets.  
dataset is larger as compared to all other 7 datasets. Table 2. clearly shows that the size of the MAMS dataset is more than double the size of other datasets. Since the performance of deep learning methods can be better evaluated on datasets of large size, thus it is pertinent to check for the performance of different deep learning methods on MAMS data.  
Our study incorporates all 8 datasets discussed above. To the best of our knowledge, no other previous study in the ALSC literature has considered 35 deep learning methods on 8 datasets.