SentiStream: A Co-Training Framework for Adaptive Online Sentiment Analysis in Evolving Data Streams
Authors: Yuhao Wu
Karthick Sharma
Chun Wei Seah
Shuhao Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: Online Sentiment analysis
Streaming learning

Abstract: Online sentiment analysis has emerged as a crucial component in numerous data-driven applications, including social media monitoring, customer feedback analysis, and online reputation management. Despite their importance, current methodologies falter in effectively managing the continuously evolving nature of data streams, largely due to their reliance on substantial, pre-existing labelled datasets. This paper presents $\textbf{sentistream}$, a novel co-training framework specifically designed for efficient sentiment analysis within dynamic data streams. Comprising unsupervised, semi-supervised, and stream merge modules,  $\textbf{ sentistream}$ guarantees constant adaptability to evolving data landscapes. This research delves into the continuous adaptation of language models for online sentiment analysis, focusing on real-world applications. Experimental evaluations using data streams derived from three benchmark sentiment analysis datasets confirm that our proposed methodology surpasses existing approaches in terms of both accuracy and computational efficiency.

SentiStream: A Co-Training Framework for Adaptive Online Sentiment Analysis in Evolving Data Streams
Yuhao Wu', Karthick Sharma?; Chun Wei Seah', Shuhao Zhang
Sinnaaore University of Technology and Design; PUniversity of Sri Jayewardenepura {yuhao_wu, chunwei_seah, Sruuha__zhang@05e edu. en938990sjp. ac.
Abstract
This necessitates continuous model adaptation to maintain effectiveness. Simultaneously, sentiment classification,
Online sentiment analysis has emerged as a crucial component in numerous data- driven applications, including social media monitoring, customer feedback analysis, and online reputation management. Despite their importance, current methodologies falter in effectively managing the continuously evolving nature of data streams, largely due to their reliance substantial, pre-existing labelled datasets. This paper presents SentiStream, novel co-training framework specifically designed for efficient sentiment analysis within dynamic data streams. Comprising unsupervised, semi-supervised, and stream merge modules, SentiStream guarantees constant adaptability to evolving data landscapes. This research delves into the continuous adaptation language models for online sentiment analysis, focusing on real-world applications. Experimental evaluations using data streams derived from five benchmark sentiment analysis datasets confirm that our proposed methodology surpasses existing approaches in terms of both accuracy and computational efficiency".
core task within
natural language processing (NLP), has found its cruciality in a myriad of sectors such as customer feedback interpretation and public opinion monitoring (Zhang et al., 2018). However, creating an OSA approach that simultaneously handle online adaptation and sentiment classification remains challenging feat.
Previous research (Vashishtha and Susan, 2019; Rahnama, 2014; Smailović et al., 2014; Go et al., 2009; Gautam and Yadav, 2014; Haque et al., 2018; Ortigosa et al., 2014) has indicated that supervised learning paradigms can yield high accuracy in sentiment analysis. Despite their strengths, these methods frequently overlook the ceaseless accumulation of real-world streaming data originating from varied sources like literature (Zhu et al., 2015), news articles (Zellers et al., 2019), and scientific papers (Lo et al., 2020). The constant emergence of this dynamic streaming data often leads to the concept drift effect, which can impair the performance of traditional offline methods (Luu et al., 2021). Continuous learning attempts to tackle concept drift and adapt the ongoing data stream, but obtaining ground truth labels for this streaming data is often arduous and costly, which consequently limits the continuous application of supervised techniques and reduces their long-term term efficacy.
1 Introduction
Online Sentiment Analysis (OSA) has established its significance in the realm of sentiment analysis, with its primary objective being the identification of polarity in ccaselessly incoming data streams (Capuano et al., 2021). This task requires proficiency in two key aspects: online adaptation and sentiment classification.
In response to these challenges, we present SentiStream, a co-training framework tailored explicitly for efficient online sentiment analysis of swift-flowing opinion data. This framework consists of three modules: unsupervised, semi- supervised, and stream merge. The unsupervised module uses continuously adapted pre-trained language models (PLMs) distill knowledge from unlabeled streaming data, coupled with sricon-based strategies to produce preliminary
The requirement for online adaptation arises from the ever-evolving characteristics of real-time data streams, an effect commonly known as concept drift (Webb et al., 2016).
"Work done while the second author was interning SUTD IntelliStream Group.
Corresponding author. !Our code is available at httpps://githu hwm. intellistreammSentiitee

---

polarity labels. SentiStream harnesses semantic and temporal information from text-based data streams to incrementally retrain the PLMs, later employing a nimble texicon-based classification method to generate polarity labels using the updated PLMs. The semi-supervised module constructs weakly supervised classification model with small labeled dataset and continuously retrains this model with pseudo-labels generated by the stream merge module. The final stream merge module consolidates outputs from the preceding two modules, utilizing their confidence scores to dynamically update the lexicon for the unsupervised module, providing pseudo-labeled data for semi-supervised learning and dynamically fine-tuning the threshold for the semi-supervised module.
2.1 Sentiment Analysis
Online sentiment analysis has gained traction with the escalating volume of user-generated content on social media platforms and online forums, MoodLens, a system developed by Zhao et al. (2012), utilizes incremental learning to navigate sentiment shifts and new terminology. In the realm of online text messages, Fernández-Gavilane et al. (2016) introduced unsupervised methodology. leveraging sentiment features from lexicons. For more comprehensive approach, losifidis and Ntoutsi (2017) employed semi-supervised learning, drawing upon both labelled and unlabeled data, via Self-Learning and Co- Training. The research in offline sentiment analysis, particularly the remarkable results attained through deep learning architectures like CNNs and RNNs (Kim, 2014; Zhang et al., 2015), and the contributions of pretrained language models such as BERT and GPT (Devlin et al., 2018; Radford et al., 2018), are also worth noting. However, these offline methodologies often falter adapting to the dynamic nature of online data streams, thereby compromising their performance. This emphasizes the need for specialized online sentiment analysis methods that can adapt fluidly to evolving data streams while maintaining performance levels.
We assessed the performance of SentiStream on five benchmark sentiment analysis datasets and juxtaposing its efficacy against several unsupervised and semi-supervised benchmarks. The experimental results confirm that our methodology considerably outperforms existing methods in tackling dynamic data streams for online sentiment analysis tasks. Additionally, SentiStream uses lightweight model, ensuring superior throughput and latency performance.
The major contributions of our work can be summarized as:
2.2 Continual Learning
The development of SentiStream, a novel co-training framework, devised specifically for proficient online sentiment analysis within dynamic data streams;
In contrast traditional neural networks, which are viewed as static knowledge entities prone catastrophic forgetting when knowledge expansion efforts veer off the original task, continual learning envisions networks capable of accruing knowledge across different tasks without requiring comprehensive retraining (De Lange et al., 2021). Prior research has primarily sought to address continual learning challenges within the frameworks of incremental class and task scenarios. The strategies used range from replay methods (Rebuffi et al., 2017; Lopez-Paz and Ranzato, 2017; Atkinson et al., 2018) and regularization-bast techniques (Kirkpatrick et al., 2017: Ahn et al., 2019) to parameter isolation methods (Xu and Zhu, 2018; Fernando et al., 2017). In noteworthy contribution, Jin et al. (2021) deployed distillation- based techniques for the continuous incremental pre-training of language models across diverse domain corpora. In the context of sentiment analysis, Ke et al. (2021) explored aspect- sentiment analysis tasks across different domains
An unsupervised module that amalgamates the merits of continuously trained PLMs with lexicon-based classification techniques;
The implementation of a ceni-supervised self- learning strategy, devised to optimize the usage limited labelled data;
The unification of outputs through stream merge technique, promoting collaborative learning to continuously adapt to dynamic stream data from various angles;
The employment of lightweight models, complemented by a series of optimizations, to fulfill online deployment requirements.
2 Related Work
In this section, we provide an overview of relevant literature on online sentiment analysis, continual learning, and semi-supervised learning, thereby laying the foundation for our proposed framework.

---

through contrastive continual learning. However, these approaches often fail address the temporal influence, as observed by Luu et al. (2021), where data drift over time can negatively impact model performance.
combat this, we employ two elements: continual pre-trained language model (PLM) training and dynamic texicon-based classifier.
Continuous PLM Training: In our framework. the PLMs are subject continuous training, a distinet departure from offline methods (Agarwal and Mittal, 2016; Haque et al., 2018) where PLMs can quickly fall behind as the vocabulary and polarity models evolve. Such offline methods necessitate periodic re-training with labelled datasets, an approach that our framework sidesteps. Our continual training ensures the PLMs remain updated, and capable of labelling sentences that contain even the most novel vocabulary.
2.3 Semi-Supervised Learning
Semi-supervised learning, which involves model building using both labelled and unlabeled data, is of particular relevance in real-world scenarios where unlabeled data is plentiful and readily available, while labelled instances are relatively scarce (Ouali et al., 2020). Internet tweets and comments are prime examples that can greatly benefit from semi-supervised learning techniques (Silva et al., 2016). These techniques comprise a variety of methods such as graph-based (Sindhwani and Melville, 2008), wrapper-based (Li et al., 2020), and topic-based (Xiang and Zhou, 2014). More recent studies have delved into dynamic thresholds for semi-supervised techniques (Sohn al.. 2020; Wang et al., 2022), with Han al. (2020) applying these methods to sentiment analysis. This research suggests an iterative auto- labelling process anchored a dynamic threshold algorithm, which takes into account both the quality and quantity of auto-labelled data when setting thresholds for their selection.
The continual PLM training module leverages rich semantic and temporal information from the streaming textual data to incrementally train the models. Specifically, the streaming data used to perpetually train the model using the pre-trained loss function in an unsupervised manner. The continuous learning aspect of our model means it can learn and refine sentence representations over time, thereby keeping in step with the evolving nature of the data stream. This unique approach allows the model to adapt more effectively to the dynamic language used in current data streams, ensuring it maintains relevance and accuracy.
Dynamic Lexicon-based Classifier: After we' ve obtained the learned vector representations of each word, we suggest leveraging text similarity measures (Wang and Dong, 2020; Vijaymeena M.K. 2016; Navigli and Martelli, 2019) to infer the sentiment polarity of an input sentence. This is based on list of reference words with pre-established polarities (Lin and He, 2009), as opposed to relying on supervised training progress for polarity identification.
3 Proposed Methodology
Our proposed method, SentiStream, consists of two parallel sentiment classification modules and a shared output co-trainer: an unsupervised module, a semi-supervised module, and stream merge module. The overall structure is shown in Figure 1. Problem Formulation: Our focus is on effectively conducting sentiment analysis of streaming opinion data real-time. We define the term imput stream as a sequence of tuples, referred to as S = Ti,...,yy which arrive at our system in chronological order. Each tuple, denoted T. consists of finite number of sentences, z. forming T =(x1
Our lexicon contains various emotion representations that we vectorize using our model. Subsequently, we conduct a cosine similarity computation (CSC) between the vectorized input and the mean of either the positive or negative reference words. The computed aggregated mean is referred as Mean(pos) Ultimately, by comparing M Mean(pos) and Mean(neg), we infer the overall polarity of the input tuple.
Jm). The sentiment
polarity is either positive or negative. Our goal with SentiStream is to learn and identify the polarity of T( ~ mm) E S as soon as T3 arrives.
or M Mean(neg).
3.1 Unsupervised Module
However, the reference table may become obsolete over time. To address this issue, we dynamically update the lexicon using sentences from the stream merge module, where the model's
Due to potential delays between training cycles, the system might not be up-to-date with events and emerging knowledge (Bubeck et al., 2023). To

---

Updating lexicon
Unsupervised
Positive
Stream
Excellent, Fantastic, Wonderful, Fun
Levicon update
Negelive
Disgust, Worst, Disappointing, Bed
Positve
Merge result
T
T
Seni-superrised
Nogative
Supervised Evaluatues
Updating semi-supervised model 04-1
Figure 1: A workflow overview of SentiStream. The unlabelled data enters both the unsupervised and semi- supervised modules, after which the results are merged and output via the stream merge module.
output confidence is high. The lexicon update algorithm, detailed in Algorithm autonomously incorporates new words into the sentiment lexicon, eliminating the need for manual annotation of new data.
Algorithm 1 Lexicon Update
Input: Sentences $ = {&1, 82... J. Initial lexicon D (Dpon: Daeg J. Similarity threshold a
Output: Updated lexicon /)' = {1) D'reg]
Compute mean embedding of positive and negative lexicon.
Mpos poosl EdeDpn." Mnee
[Pacyl EdeDans
We establish a similarity threshold that offers a balance between the need to add new words and the risk of adding false positives to the reference table. Hence, new words only added to the reference table if the cosine similarity between the lexicon and the word surpasses the similarity threshold.
for each sentence in do
for each word wr s
Calculate cosine similarity between w and poos if cos( w. (fpos) > a then Add w to Djos end if
Calculate cosine similarity between w and parg if cos( (w. (kmeg] >> a then Add w to D'eeg end if end for end for
Through this dynamic update process, our lexicon continuously reflects the prevailing ways emotions are expressed in everyday language. Our extensive experimental results demonstrate that this approach not only significantly reduces computational effort but also consistently outperforms alternative methods in terms of prediction accuracy.
return D' = (Dpoee D'neg)
accurate and consistent pseudo-labels challenging task. If changes occur within the dataset, it may be difficult for weakly supervised models make accurate decisions. Likewise, static thresholds may not yield an adequate number of pseudo-labelled data under these changing conditions.
3.2 Semi-Supervised Module
The semi-supervised module's primary goal is to efficiently utilize limited amount of labeled data alongside a substantial amount of unlabeled data obtained from the streaming input, thereby promoting semi-supervised learning. To accomplish this, we use labelled data collected from the stream merge module to identify instances with high confidence, which then used as pseudo-labels for the continual training of the semi- supervised classifier.
Dynamic Threshold: In order to overcome these challenges, we employ dynamic threshold approach, as proposed in the study by Zhang et al. (2021). As shown in Algorithm 2, this method adjusts the threshold for each class based on the model's current learning status. The learning efficiency of class assessed by counting the number of samples whose predictions surpass the hard threshold, defined as the sum of the lower and
However, the ever-evolving nature of streaming data environments can make the acquisition of

---

upper thresholds. This count is then normalized by the maximum value of either the positive or negative learning effect or the number of low- confidence labels. This approach is especially useful in the early learning stages when the learning effect is generally minimal, and a higher number of low-confidence labels would naturally lead to more flexible threshold.
Algorithm 3 Stream Merge
Input: Unsupervised model's predicted labels U; Ut Ui.. Unsupervised model's predicted confidence U. = Ue Ue.. Semi-supervised model's predicted labels Si = Si.... St..... Semi-supervised model's predicted confidence S. = Seg....,,S Fixed confidence threshold T', Adaptive weight for unsupervised prediction W.. Adaptive weight for semi-supervised prediction W, Output: Predictions P' = y p.
for i + to do if Ue; > T and Se: > T then if Ug. > Sc, then
Following this, we utilize a non-linear function to adjust the learning rate. Initially, this function incrementally increases the threshold, but quickens the rise when both learning rates are high. This method allows for more scamless and logical integration of data, thereby improving the quality of the produced pseudo labelled data.
P Ui,
else
p" St.
end if else
if Ue.
W. > Sc. x W. then
pr Ui,
else P' St
Algorithm 2 Dynamic Threshold
end if
end
Input: Pseudo labels P = fpi....Ppi Confidence scores C' (c),...c,): Learning effect À = (Doon Anegl- Fixed lower threshold a = {Apos, Onegh. Fixed upper threshold B = {[3pna, 8neg J-
end for W..
Ei 2..T
W, D 2e
Output: Filtered pscudo labels p = (p'....D''
return P' [ph....p'.
pos Eepaportipas
neg- + Las-lannt
if pos + ney > then
ô max((C) (pos + neg), pos, neg)
models' performance over time. For every data point, the algorithm checks whether both models show high confidence in their predictions (surpassing a threshold T). If so, the prediction associated with the highest confidence score is selected. Otherwise, the algorithm multiplies the confidence scores by the adaptive weights for cach model, and the prediction with the highest weighted confidence score is chosen.
Apos (pos/8) (2 po)/6)
Anng~ (neg/ /2 neg/
end if
c +-++1
to do
if e -(aneg + Bneg )neg) c. (Oipos Spos
then
p' Pa
end if
end
return p' (ph....pppa.
This approach prioritizes labels with high confidence, which generally lead correct labels, while also addressing the issue of inaccurate low-confidence labels. By multiplying the model's weight with low-confidence predictions, the algorithm ensures that even when predictions have low confidence, the highest-performing model is given more importance. This, in turn, contributes to making the overall prediction process more consistent and logical.
3.3 Stream Merge Module
The principal aim of the stream merge module is to competently amalgamate data and yield reliable pseudo-labelled data. To achieve this, we introduce stream merge method hinging confidence assessment. As depicted in Algorithm 3. this method merges data streams generated by two separate parts, selecting data points accurately classified (with high confidence) by both models to form lleuds-labels.
Experimental Setup
The algorithm dynamically adjusts the weights for each model, basing its decision on its model
4.1 Datasets
In conducting our evaluation, employed two distinct types of datasets to thoroughly assess the adaptability of SentiStream to dynamic data streams, each labeled according to specific classification rules. These datasets are delincated as follows:
previous prediction performance. demonstrating higher ratio of high-confidence predictions will be allocated more weight in the subsequent iteration, thereby potentially improving its prediction accuracy while adapting to the

---

on Amazon. These reviews were collected in chronological order in quarterly periods from 2010 to 2018.
SST-2
LMRD
Yelp
4.2 Language Model Set
In this paper, we select some lightweight language model to achieve lower latency and higher throughput, in line with the industry's deployment preference for simple and efficient models. Galke and Scherp (2021) showed that combining a bag-of-words model with WideMLP resulted in exceptional performance in text classification tasks. Characterized by a single wide hidden layer, WideMLP outperforms numerous contemporary models in inductive text categorization. Furthermore, training large language models (e.g. BERT, GPT) on streaming data involves additional complexities and constraints, which we will discuss in Section 7. Therefore, here we choose Word2Vec (Mikolov et al., 2013) and Hierarchical Attention Networks (Yang et al.. 2016) as the base models for unsupervised and semi-supervised learning, respectively, We also include BERT (Devlin et al., 2018) large-scale model for comparative testing.
Figure 2: Data stream (Yelp - LMRD - SST-2)
4.1.1 Multi-domain Evolving Datasets
Formulated by amalgamating three well-known, large-scale datasets, this construction simulates dynamic data stream featuring evolving characteristics, thereby reflecting real-world data fluctuations across multiple domains and temporal spans.
Yelp Review Polarity (Zhang et al., 2015) was derived from the Yelp Dataset Challenge in 2015. Reviews were labeled as either positive, if they received 3 or 4 stars, negative, if they received stars.
Large Movie Review (LMRD) (Maas et al., 201 was collected by the Artificial Intelligence Laboratory Stanford University. This dataset contains movic reviews along with their associated binary sentiment polarity labels, serving as a benchmark for sentiment classification.
Stanford Sentiment Treebank-2 (SST- 2) (Socher et al., 2013) collected by Stanford University researchers. It consists of movie reviews extracted from Rotten Tomatoes parsed using Stanford parser with sentiment labels.
4.3 Baselines
We compare the performance of the proposed model with diverse types of baselines such as random, supervised and self-super methods.
We evenly sample data from the three datasets, merging them in three different orders (1, Yelp LMRD - SST-2; 2,LMRD -> SST-2 -> Yelp; 3, SST-2 + Yelp - LMRD) to simulate real-world data drift, with one of these scenarios illustrated in Figure 2.
Random: At first, we present random baseline where the predictions are generated using uniform distribution. This will provide with lower bound for evaluation.
Supervised: We train supervised model by using 0.5% of the entire dataset as the training set. We chose BERT and HAN as the two basc models for our experiments. This will provide us with an upper bound for our evaluation.
4.1.2 Longitudinal Singular Domain Datasets Incorporates two datasets with longitudinal perspective within a single, consistent domain, providing insight into how SentiStream handles changes and shifts over time within the same data source
Self-supervised:
The self-supervised framework used by (losifidis and Ntoutsi, 2017) employed co-training to improve the model performance.
Sentiment140 (Go et al., 2009) was compiled using the Twitter API, encompassing balanced distribution of 1.6 million tweets, expressed sentiments, and recorded in chronological order from April 6, 2009, to June 25, 2009.
We select weakly- supervised framework (CL-WSTC (Li et al., 2023)) that considers the scenario of continual learning for comparison. They employ BERT as the foundational model.
Weakly-supervised:
Amazon Fashion, a subset of the Amazon Review Data (Ni et al., 2019) and consists of customer reviews for fashion products available

---

5 Experimental Evaluation
Method
Latency(ms)
Throughput
Self-supervised
2.43
409
this section, we study the performance of the different algorithms on three datasets, compare them with different baselines, and discuss the qualitative analysis of our model's performance.
Weakly-supervise
584.75
Supervised (BERT)
53.17
162
SentiStream
0.67
Table 1: End-to-end evaluation of the throughput and latency
5.1 Evaluation Framework
5.3 Experimental Results and Analysis
We evaluate our framework within an end-to- end setup for real-time sentiment classification, with throughput, latency, accuracy, and streaming data adaptation serving as primary performance indicators. The task requires processing text data within specific time intervals and assigning the appropriate emotion labels. To verify the effectiveness of our framework, we will run experiments on integrated data streams. We
5.3.1 End-to-end comparison
Latency and Throughput: In Table 1, we can see that SentiStream surpasses self-supervised and weakly-supervised methods, in terms of latency and throughput. The BERT model's latency nearly 100 times that of SentiStream. Moreover, the difference in throughput is especially pronounced, with SentiStres outperforming the BERT model by nearly an order of magnitude. This discrepancy can be ascribed to the stark contrast in the number of model parameters utilized by SentiStream compared to those used by the BERT model. As for the weakly-superviser method, it was initially developed without considering latency and throughput. Consequently, its approach tends to prioritize future accuracy enhancements, even if potentially compromises latency and throughput.
designate 0.5% of the ongoing data stream as training data for our semi-supervised model and other supervised model, while the remaining portion is utilized as test data. This enables us to assess the performance at each stage and the overall performance. An ablation study provides insights into the unique effects of various optimizations, but due to space constraints, this has been relocated to Appendix A.I.
5.3.2 Multi-domain Evolving Datasets
5.2 Evaluation Metrics
Overall Performance: Table 2 presents a comprehensive summary of our experimental results. Our framework, SentiStream, consistently exhibits excellent performance, occasionally even matching the supervised BERT baseline. In addition, the sentistream approach is always excellent with the supervised HAN and word2vec models. In terms of the Fl score, SentiStream substantially outperforms its counterparts, thereby showcasing the power its co-training strategy for handling unbalanced data.
We evaluate the system with five performance metrics.
Throughput. High throughput is necessity to manage large-volume data streams. For instance, in the event of significant happenings, opinions on social media may suddenly surge. Thus, we measure throughput as the maximum number of input tuples per second that the system can sustain.
Latency. We measure the 95% latency as the elapsed time from when the input tuple arrives to when the corresponding classification result is produced. It is an important indicator to denote the system's responsiveness.
The performance of the unsupervised component is particularly noteworthy, underscoring the bag- of words model's ability to produce satisfactory results, even in less complicated text classification tasks. As the data suggests, SentiStreat performance improves with extended offsets.
Accuracy. We define prediction accuracy as the ratio of correct predictions (the sum of true positives and true negatives) the total number of tuples processed.
Figure 3 presents the dynamic performance of accuracy across the entire datasets, assessing the continual learning capability our method. Initially, during the Yelp dataset stage, supervised learning shows
Adaptation to Streaming Data:
F1-score & AUC. To evaluate the accuracy of the prediction workload with class imbalance, we also use the Fc-score and AUC, which is the harmonic mean of precision and recall.

---

Method
Yclp(%)
LMRD(%)
SST-2(%%
Total (6)
F1
AUC
Random
49.68 %
50.12
50.07 %
50.16 %
51.07 %
50.19
Self-supervised
64.74 %
49.81 %
44.77 %
57.07 %
45.74 %
47.66 %
Weakly-supervised
66.11 %
51.97% %
40.91 %
52.94 %
42.19 %
45.11
Supervised (W2V)
75.42 %
60.64 %
57.65 %
65.79 %
65.00 %
73.44 %
Supervised (HAN)
75.38 %
65.91 %
50.24 %
63.01 %
68.13 %
74.85
Supervised (BERT)
86.07 %
78.19 %
46.48 %
70.25 %
72.60 %
78.99 %
Us module [3.1]
81.60 %
77.41 %
76.23 %
78.79 %
79.01 %
82.84
Ss module [3.2]
79.80%
68.63 %
73.88 %
74.98 %
70.49 %
82.20 %
SentiStream 81.73 %
77.36
76.22 %
78.82 %
79.02 %
84.97
Table 2: Yelp, LMRD, and SST-2 correspond the three parts of the merged dataset, Total indicates the average performance of the model in the three orders. F1 refers to the overall F1 score, and the average latency and throughput are also provided. For different methods, the first three are the baselines we have listed in section 4.3. followed by the results for the unsupervised module, semi-supervised module, and our framework in section 3.
remarkable performance, primarily due to the powerful capabilities intrinsic to BERT. However, when concept drift occurs, supervised learning fails to adapt and learn continuously (as it requires annotated data for model fine-tuning), leading to decline in performance. This downward trend in performance becomes significantly evident during the final task involving the SST-2 dataset.
9.8-
Accuracy
6.6-
Saf superises Suporvsolieeet
0.5
SentiSmaa
5545556
Figure 3: Performance in Multi-domain Stream data, is Yelp dataset, L is LMRD dataset, is SST dataset. Each dataset has six sequential accuracy values, representing the progressive adaptation in the stream data.
Surprisingly, methods expected to demonstrate substantial continual learning capabilities, such self-supervised learning and weakly supervised learning, did not meet the anticipated performance metrics. The underlying model for self-supervised learning is Bayesian, which might be too simple to effectively mine and extract valuable information. Upon closer examination of weakly supervised learning, a major issue was identified: the model's failure to properly integrate seed words during its operation in the online sentiment classification task. This problem is critical given the significant influence these seed words have the model's s performance.
5.3.3 Longitudinal Singular Domain Datasets
Overall Performance: With regard to the aggregate performance. SentiStream exhibits outstanding proficiency both the Sentiment140 and Amazon Fashion datasets. Notably, SentiStream outperforms its rivals in metrics such ACC, Fl-score, and AUC.
As illustrated in Figures 4 and there are notable performance shifts over time for the self-supervised, supervised. and SentiStrean methodologies. the context of Figure 4, SentiStream markedly distinguishes itself from the other two approaches. On the other hand, in Figure 5, early self-supervised and supervised methods initially lead the pack. However, as the data distribution shifted, for instance during the 2013-Q3 quarter, SentiStream leveraged its superior adaptability and took a comprehensive lead. In summary. SentiStream showcases commendable average performance
Adaptation to Streaming Data:
the contrary, SentiStream displays exceptional performance, consistently adapting to the evolving data distribution inherent to the data stream, thus promoting continuous performance improvement. A prime example is SentiStreat 's progressively improving performance throughout the SST dataset. It also demonstrated notable resilience in dealing with data drift. Such adaptability particularly important real-world scenarios where concept drift can occur in more complex forms.

---

Dataset
Method
ACC(%%
F1
AUC
Self-supervised
63.75 %
63.07
66.26 %
Weakly-supervised
53.51 %
52.71
49.23 %
Supervised (W2V)
66.09 %
65.78 %
60.31 %
Sentiment140
Supervised (HAN)
64.87 %
68.91 %
61.88 %
Supervised (BERT)
59.90 %
66.58 %
61.96 %
SentiStream
67.81 %
67.56 %
72.21 %
Self-supervised
78.03 %
87.65
52.07 %
Wcakly-supervised
64.27 %
51.17 %
49.95 %
Supervised (W2V)
78.11 %
87.71 %
64.56 %
Amazon Fashion
Supervised (HAN)
74.81 %
80.12 %
69.39 %
Supervised (BERT)
76.01 %
82.71 %
50.00 %
SentiStream
85.47 %
90.03
93.50 %
Table 3: Performance comparison of different methods for Sentiment1 and Amazon (Fashion).
and robust adaptability to streaming data in Longitudinal Singular Domain Datasets.
cannot be trained on streaming data, presenting various challenges, such as computational resource utilization, real-time updates, data instability, hyperparameter tuning, storage and management, and system stability and reliability. Bubeck et al. (2023) also emphasizes the importance of continuous learning for LLM and indicates the need for further research and improvement, while our work can seen as an initial, yet important attempt towards such goal. Another practical issue is the absence of publicly available corpora categorized by domain or year, along with corresponding sentiment classification test sets. Moving forward, aim address this limitation by working with genuine large-scale language models trained on streaming data, showcasing their effectiveness in more complex tasks.
6 Conclusion
This paper introduced SentiStream, an adaptive co-training framework that efficiently tackles concept drift, latency, and throughput issues in dynamic data streams for online sentiment analysis. Through its integrated unsupervised, semi-supervised, and stream merge modules, SentiStream effectively manages continuous data stream evolution, a major challenge for existing methods. Continuous training and dynamic dictionary updates enhance SentiStream's adaptability to ever-changing data streams, proving its potential applicability in real-world scenarios. Experimental results demonstrated SentiStream's promising performance in online sentiment analysis across various data-driven applications. As highly adaptable and efficient solution, SentiStream addresses the growing demand for real-time sentiment analysis in evolving online environments and data streams. Future work can build on this foundation, extending the application of the SentiStream framework to other dynamic, data-driven domains.
Acknowledgement
We would like to thank the anonymous reviewers. meta-reviewer, and senior chairs for their insightful comments and support with this work. This project is supported by TL@SUTD seed research project grant (RTDSS2214051) and the National Research Foundation, Singapore and Infocomm Media Development Authority under its Future Communications Research & Development Programme FCP-SUTD-RG- 2022-006. The computational work for this article was partially performed on resources of the National Supercomputing Centre (NSCC), Singapore (httss//www.nnscc.s)).
7 Limitation
A primary limitation of our study is the lack of integration with popular large-scale language models (e.g., GPT4 (OpenAI, 2023)). However, is worth noting that employing these models entails increased computational resources and latency. Moreover, current large-scale language models

---

References
Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter sentiment classification using distant supervision. CS224N project report, Stanford, 1(12):2009.
B. Agarwal and N. Mittal. 2016. Machine Learning Approach for Sentiment Analysis. Springer International Publishing.
Yue Han, Yuhong Liu, and Zhigang Jin. 2020. Sentiment analysis via semi-supervised learning: a model based on dynamic threshold and multi- classifiers. Neural Computing and Applications, 32:5117-5129.
Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon. 2019. Uncertainty-based continual learning with adaptive regularization. Advances in neural information processing systems,
Tanjim Haque, Nudrat Nawal Saber, and Faisal Muhammad Shah. 2018. Sentiment analysis on large scale amazon product reviews. In 2018 IEEE international conference on innovative research and development (ICIRD), pages 1 1-6, IEEE
Craig Atkinson, Brendan McCane, Lech Szymanski, and Anthony Robins. 2018. geeudo-recusaal Solving the catastrophic forgetting problem in deep neural networks. arXiv preprint a80i:: 03875,
Vasileios losifidis and Eirini Ntoutsi. 2017. Large scale sentiment learning with limited labels. In KDD' 17, pages 1832.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee. Yin Tat Lee, Yuanzhi Li, Scott Lundberg. et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint 1.Xiv2333.122122
Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. 2021. Lifelong pretraining: Continually adapting language models to emerging corpora. arXiv preprint urXiv:2110.085::
Nicola Capuano, Luca Greco, Pierluigi Ritrovato, and Mario Vento. 2021. Sentiment analysis for customer relationship management: an incremental learning approach. Applied Intelligence, 51(6):3339-333 3352.
Zixuan Ke, Bing Liu, Hu Xu, and Lei Shu. 2021. Classic: Continual and contrastive learning of aspect sentiment classification tasks. arXiv preprint arXiv:2112.02714.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, Tinne Tuytelaars. 2021. continual learning survey: Defying forgetting in classification tasks. TEFE transactions on pattern analysis and machine intelligence, 44(7):3366-1 3385.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 conference on empirical methods in natural language processing (FMNLP), pages 1746-1751.
Jacob Devlin, Ming-War Wei Chang, Kenton Lec, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, et al. 2017. Agnieszka catastrophic forgetting in neural Overcoming networks. Proceedings of the national academy of sciences, 114(13):55113355
Milagros Fernández- López, Jonathan Ameal--arttneed Enrique Costa- Montenegro, and Francisco Javier González-Castal 2016. Unsupervised method for sentiment analysis online texts. Expert Systems with Applications. 58:57-75.
Tamara Álvarez-
Miaomiao Li, Jiaqi Zhu, Xin Yang, Yi Yang, Qiang Gao, and Hongan Wang. 2023, Cl-wste: Continual learning for weakly supervised text classification on the internet. In Proceedings of the ACM Web Conference 2023, pages 1489-1499,
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. 2017. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint
Ning Li, Chi-Yin Chow, and Jia-Dong Zhang. 2020. Seml: semi-supervised multi-task learning framework for aspect-based sentiment analysis. IEEE Access, 8:189287-189297.
Lukas Galke and Ansgar Scherp. 2021. Bag-of- words vs. graph vs. sequence in text classification: Questioning the necessity of text-graphs and the surprising strength of wide mlp. arXiv preprint arXiv:2109.03777.
Chenghua Lin and Yulan He. 2009. Joint sentiment/topic model for sentiment analysis. Proceedings of the 18th ACM conference on Information and knowledge management, pages 375- 384.
Geetika Gautam and Divakar Yadav. 2014. Sentiment analysis of twitter data using machine learning approaches and semantic analysis. In 2014 Seventh International Conference on Contemporary Computing (IC3), pages 437-442, IEBE.
Kyle Lo. Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of 58th Annual Meeting of the Association for Computational Linguistics, pages

---

4969-4983, Online. Association for Computational Linguistics.
Vikas Sindhwani and Prem Melville. 2008. Document- word co co-regularization for supervise sentiment analysis. In 2008 Fighth ieee international conference on data mining, pages 1025-1030. IEEE,
David Lopez-Paz and Mare' Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30.
Jasmina Smailovié. Miha Grëar, Nada Lavraë, and Martin Znidaršič. 2014. Stream-based active learning for sentiment analysis in the financial domain. Information sciences, 285:1.
Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Karishma Mandyam, and Noah A Smith. 2021. Time waits for no one! analysis and challenges of temporal misalignment. arXiv preprint vrXiv:2111.07400
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng. and Christopher Potts. 2013, Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 163 - 1642.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Linguistics.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang. Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. 2020. Fixmatch: Simplifying remi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-60 6088.
Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Efficientt estimation of word representations in vector space. In ICLR.
Srishti Vashishtha and Seba Susan. 2019. Fuzzy rule based unsupervised sentiment analysis from social media posts. Expert Systems with Applications. 138: 1 12834.
Roberto Navigli and Federico Martelli, 2019. An overview word and sense similarity. Natural Language Engineering, 25(6):693-714.
Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. Justifying recommendations using distantlyllbelld reviews and fine-grate ine-grained aspects. pages 188-197.
Kavitha K Vijaymeena M.K. 2016. A survey similarity measures in text mining. Machine Learning and Applications: An International Journal, 3(1):19-28.
OpenAI. 2023. Gpt-4 technical report.
Jiapeng Wang and Yihong Dong. 2020. Measurement of text similarity: A survey. Information, 11(9).
Alvaro Ortigosa, José M Martín, and Rosa M Carro. 2014. Sentiment analysis in facebook and its application to e-learning. Computers in human behavior, 31:527-541.
Yidong Wang, Hao Chen, Qiang Heng. Wenxin Hou, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj. Zhen Wu, and Jindong Wang. 2022. Freematch: Self- adaptive thresholding for semi- superviid learning. arXiv preprint arXiv:2005.07246.
Yassine Ouali, Céline Hudelot, and Myriam Tami. 2020. An overview deep supervier learning, arXiv preprint arXiv:2006.05: 05578..
Geoffrey Webb. Roy Hyde. Hong Cao.
Alec Radford,
Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training.
Hai Long Nguyen, and Francois Petitjean. 2016. Characterizing concept drift. Data Mining and Knowledge Discovery, 30(4):964-994.
Amir Hossein Akhavan Rahnama. 2014. Distributed real-time sentiment analysis for big data social streams. In 2014 International conference control, decision and information technologies (CoDIT), pages 789-794. IEEE
Bing Xiang and Liang Zhou. 2014. Improving twitter sentiment analysis with topic-based mixture modeling and supervised training.
Proceedings of the 52nd Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pages 434 439.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pages 2001-2010.
Ju Xu and Zhanxing Zhu. 2018. Reinforced continual learning. Advances in Neural Information Processing Systems, 31.
Zichao Yang. Diyi Yang, Chris Dyer, Xiaodong He. Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human
Nadia Felix Da Silva, Luiz FS Coletta, and Eduardo Hruschka. 2016. A survey and comparative study of tweet sentiment analysis via semi-supervised learning. ACM Computing Surveys (CSUR), 49 26.

---

Language Technologies, pages 1480-1489, San Diego, California. Association for Computational Linguistics.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. 2021. Flexmatch: Boosting semi- supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 34:180 14488-188811
Lei Zhang, Shuai Wang. and Bing Liu. 2018. Deep learning for sentiment analysis: survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8()):12233
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level Convolutional Networks Text Classification arXtv:1509.01622 [cs].
Jichang Zhao, Li Dong, Junjie Wu, and Ke Xu. 2012. Moodlens: emoticon-based sentiment analysis system for chinese tweets. Proceedings of 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1528- 1531.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov. Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference Computer Vision (ICCV), pages 19-27.

---

A Appendix
update and dynamic threshold in the baseline model. This comprchensive approach aimed to leverage the benefits of both techniques simultaneously. For this variation, we fixed the similarity threshold at 0.9 and the upper threshold at 0.8.
A.1 Ablation Study
To assess the contribution of each proposed method in improving the performance of our semi-supervised learning model, we conducted an ablation study, as shown in Table 7. The study aimed to evaluate the contributions of dynamic lexicon update and dynamic threshold, as well as their combination, in enhancing sentiment analysis performance. The study consisted of four model variations, namely the baseline model, the model with dynamic lexicon update, the model with dynamic threshold, and the model combining both dynamic lexicon update and dynamic threshold. By conducting this experiment, we sought to provide empirical evidence regarding the effectiveness of these methods and their impact on sentiment analysis performance. Understanding the contributions of dynamii lexicon update and dynamic threshold, both individually and in combination, can guide the development of more accurate and robust sentiment analysis models, especially in scenarios with limited labeled data availability.
The results clearly demonstrate that the combined model outperforms the other variations, indicating the effectiveness of leveraging both dynamic lexicon update and dynamic thresholding for improved sentiment analysis performance.
A.2 Device Setting
This is information about the device currently used in the experiment.
CPU
17-13700ks
Memory
64GB
GPU
A6000
Table 4: The table shows the device information.
A.3 Word List
Table 5 shows the initial word list, used Algorithm
The baseline model served as starting point for comparison. It employed a fixed threshold method with the upper threshold set to 0.8 for filtering pseudo labels and did not incorporate dynamic lexicon update or dynamic threshold.
Baseline Model:
brilliant bliss excellent fantastic super masterpiece admire cool amuse love wonderful best great rejoice bcautiful awesome fun terrible awful unwatchable bad disgust boring stupid bullshit abuse outrage rubbish wonst awkward disappointing fraud
Positive
Model with Dynamic Lexicon Update: In this variation, we introduced dynamic lexicon update the baseline model while maintaining fixed threshold for pscudo label filtering. is aimed to leverage an evolving lexicon to enhance sentiment classification. Similarity threshold (a) was varied to explore its impact on the model's performance. Specifically, we experimented with three values of 0.7, 0.8 and 0.9.
Negative
Table Reference table used in our experiments
A.4 Distribution of Sentiments in Datasets
Dataset
Negative
Positive
Model with Dynamic Threshold: This variation incorporated dynamic thresholding into the baseline model, but did not involve lexicon updates. allowed for adaptively adjusting the threshold for pseudo label filtering based on the model's predictions. We examined the effects of different upper threshold values (T) on the model's performance. The same three values of T (0.7, 0.8. and 0.9) used in the previous variation were employed.
Yelp
40227
39773
LMRD
24698
24884
SST-2
30076
37779
Sentiment140
800000
800000
Amazon Fashion
170924
610225
Table 6: Sentiment distribution in datasets
A.5
Additional Results
Model with Combined Dynamic Lexicon Update and Dynamic Threshold: In the final variation, we combined both dynamic lexicon
A.5.1 More Training Data Evaluation Results
We used more as (1%) the total training data, although this is probably the more rare case. In the

---

Baseline
Lexicon Update
Dynamic Threshold
Final
0=0.7
x=0.8
a=0.9
T=0.7
T=0.8
T=0.9
acc
72.57 %
61.13 %
72.65 %
78,20
71.56 %
72.40 %
72.45
78.83 %
Us module
fl
73.67 %
68.76 %
75.87 %
78.91
72.42 %
73.45 %
73.59
79.66 %
Ss module
acc
72.13 %
68.44 %
72.13 %
74.18 %
71.26 %
72.92 %
69.36 %
74.90 %
69.36 %
63.70 %
69.96 %
71.25
63.61 %
72.57 %
64.87
71.99 %
acc
72.95 %
64.82 %
73.99 %
77.06 %
70.98 %
73.75 %
70.20 %
78,94 %
SentiStream
71.33
70.46
76.54
77.70
71.24 %
72.88 %
66.48
79.60 %
Table 7: Ablation study of different model variations. The table shows the accuracy and Fl score achieved by cach variation under different experimental conditions, such as lexicon update with varying similarity threshold (a) and dynamic threshold with varying upper threshold (T).
Method
Yelp(%)
LMRD(%)
SST-2(%%
Total( (%)
AUC
Latency(ms)
Throughput
Random
49.88
50,45 %
50.26
50.10
50,94 %
50.06
Self-supervised
68.92 %
49.83 %
44.69 %
55.66 %
32.19 %
46.81 %
2.42
480
Supervised
90.82 %
81.83 %
65.98
79.55
79.88
87.83
54.85
155
Us module [3.1]
83.41 %
76.56 %
76.16 %
79.16 %
80.14 %
82.96 %
Ss module [3.2]
82.19%
68.10 %
74.00 %
75.77 %
75.89 %
82.22 %
SentiStream 83.50 %
76.52 %
76.00 %
79.12 %
80.06 %
84.98 %
0.77
1292
Table 8: The primary distinction between the current table and Table 2 lies the volume of training data utilized: the present table incorporates 1% training data, albeit such instances uncommon. practical settings, acquiring a substantial amount of labeled data for training, particularly within streaming contexts, is problematic. Real-time manual labeling is virtually infeasible.
results, the supervised effect of BERT is better due to having more training data, but still inevitably followed by significant drop the SST-2 dataset indicated in table 8.
239
eve
A.5.2 Multi-domain Evolving Datasets
We conducted comprehensive experiments involving various combinations of datasets to evaluate the performance of our method compared to different baselines. Notably, our method consistently outperformed the alternative baselines in all combinations. Specifically, we examined the Yelp + LMRD SST-2, LMRD + SST-2 + Yelp and SST-2 Yelp LMRD combinations. as illustrated in Table 9, Table 10, Table 11 respectively. The superior effectiveness of our method in addressing the research problem is demonstrated by its consistent performance across various dataset arrangements.
Figure 4: Performance in Longitudinal Stream data
(Sentiment 140).
Acturacy
10000
200
Figure 5: Performance in Longitudinal Stream data (Amazon Fashion).
A.5.3 Longitudinal Singular Domain Datasets
140 and Amazon Fashion. The SentiStream demonstrates notable average performance and robust adaptability to streaming data within these datasets.
Table 12 and Table 13 present the ACC, F1, AUC, throughput, and latency metries for longitudinal singular domain datasets. Figure and 5 depict the temporal trends of supervised, semi-supervised, and SentiStream performance on Longitudinal Singular Domain Datasets, specifically Sentiment

---

Method
Yelp(%)
LMRD(%)
SST-2((%)
())
AUC
Random
49.68 %
50.12 %
50.07 %
50.16 %
51.07 %
50.19 %
Self-supervised
64.74 %
49.81
44.77 %
57.07 %
45.74
47.66
Weakly-superrvsed
66. %
51.97 %
40.91 %
52.94 %
42.19 %
45.11 %
Supervised (W2V)
75.42 %
60.64 %
57.65 %
65.79 %
65.00 %
73.44 %
Supervised (HAN)
73.38 %
65.91
50.24
63.01 %
68.13
74.85
Supervised (BERT)
86.07 %
78.19 %
46,48 %
70.25 %
72.60 %
78.99 %
Us module [3.1]
81.60 %
77.41
76.23 %
78.79 %
79.01 %
82.84 %
Ss module [3.2]
79.80%
68.63
73.88 %
74.98 %
70.49 %
82.20 %
SentiStream
81.73 %
77.36 %
76.22 %
78.82 %
79.02 %
84.97 %
Table Performance comparison of different baselines for Yelp + LMRD SST-2 combination.
Method
LMRD(%)
SST-2(%)
Yelp(%)
Total(%)
AUC
Self-supervised
73.95
69.08 %
50.83%
62.85 %
53.43
43.78
weperi-superviied
51.48
51.37 %
43.91 %
47.59 %
44.55 %
43.61 %
Supervised (W2V)
73.79 %
62.11 %
72.18 %
69.02 %
69.45 %
74.75 %
Supervised (HAN)
73.14
56.91 %
74.64%
70.42 %
69.16
77.24
Supervised (BERT)
80.42 %
56.17 %
83,49 %
73.36 %
70.16 %
90.73 %
Us module [3.1]
76.59
75.69 %
83.02 %
78.49 %
79.83 %
82.66 %
Ss module [3.2]
74.48%
74.97 %
79.86
76.84 %
78.32 %
84.03 %
SentiStream
76.63 %
76.76 %
83.05 %
79.34 %
80.19 %
86.26 %
Table 10: Performance comparison of different baselines for LMRD SST-2
Yelp combination.
Method
SST-2(%)
Yelp(%)
LMRD(%)
Total(%)
AUC
Self-supervised
68.43 %
49.81
50. 12%
56.23 %
63.35
48.80
Weakly-supervised
55.19 %
45.57 %
49.13 %
49.92 %
45.64 %
45.98 %
Supervised (W2V)
55.69 %
49.72 %
50.15 %
51.91 %
68.35 %
50.74 %
Supervised (HAN)
53.96
49.81
48.67%
49.99 %
68.75
49.79
Supervised (BERT)
72.30 %
60.81 %
66.12 %
66.41 %
71.48 %
63.58 %
Us module [3. 1]
61.13 %
83.82 %
81.27 %
75.40 %
76.72 %
75.32 %
Ss module [3.2]
53.21
71.23
67.26
64.04 %
58.34
75.18
SentiStream
61.13 %
83.88 %
82.71 %
75.91 %
76.73 %
79.38 %
Table 11: Performance compaarssn of different baselines SST-2 Yelp -> LMRD combination.
Method
ACC(%)
AUC
Latency(ms)
Throughput(tulle/s))
Random
49.95
40.79
49.95
Self-supervised
63.75 %
63.07 %
66.26 %
3.57
412
weakly-supervised
53.51 %
52.71 %
49.23 %
419.61
19
Supervised (W2V)
66.09
65.78
60.31 %
0.01
54276
Supervised (HAN)
64.87 %
68.91 %
61.88 %
0.03
17853
Supervised (BERT)
59.90
66.58
61.96
36.90
Us module
67.73 %
66.46 %
69.05 %
module
64.46
67.96
71.51
SentiStream
67.81
67.56
72.21 %
0.21
4518
Table 12: Performance comparison of different baselines for Sentiment 40.
Method
ACC(%)
AUC
Latency(ms)
Throughput(uppee/s)
Random
50.06
61.02
50.07
Self-supervised
78.03 %
87.65
52.07 %
2.20
453
weakly-supervised
64.27
51.17
49.95 %
538.90
Supervised (W2V)
78.11
87.71
64.56 %
0.02
43447
Superviscd (HAN)
74.81 %
80.12 %
69.39 %
0.03
15132
Supervised (BERT)
76.01
82.71
50.00 %
46.07
227
Us module [3.1]
83.02 %
89.18 %
87.17 %
module [3.2]
82.93
86.86
92.29
SentiStream
85.47 %
90.03 %
93.50 %
0.28
3511
Table 13: Performance comparison of different baselines for Amazon (Fashion).


Reviews:
[{"review_id": "k7xvxqGnJn", "reviewer": "Reviewer_11Lv", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Authors' Response Letter** to the Reviewers of the Manuscript:\n\nSentiStream: A Co-Training Framework for Adaptive Online Sentiment Analysis in Evolving Data Streams \n\nThanks again for the detailed comments and constructive feedback. Below we discuss how we have addressed each comment as reflected in the revised manuscript. To ease your review, your question will be **bolded**.\n\n**Q1: Empirical results show that the unsupervised module is the major source of improvement in the performance. The integration of the semi-supervised module can only slightly improve, or even degrade the overall performance. It cannot justify the need for the semi-supervised module in the framework. In particular, without the semi-supervised module, the framework does not need any training examples. Following this, the proposed framework employs a lightweight language model in the unsupervised module. If a more sophisticated language model is used, it is unclear if the semi-supervised module is still needed.**\n\nWe appreciate the reviewer's insightful comment on the role of the semi-supervised module in our framework. We acknowledge that the unsupervised module is a significant contributor to performance. We would like to clarify the following points:\n\n- *Online Setting Adaptability:* Large language models, even when fine-tuned, may not adapt swiftly enough to rapidly changing data streams (Ghunaim et al. 2023). Our semi-supervised module is designed to mitigate this limitation by leveraging a small amount of labeled data for quicker adaptability.\n- *Initial Phase Performance:* During the initial phase, where the unsupervised module has not yet adapted to the data stream, the semi-supervised module demonstrates superior performance. This is crucial for applications where immediate accuracy is imperative. for example,  such as the continuous emergence of new domains (concept drift occurs more frequently and more severely), Ss module can exert a greater impact due to its strong initial performance. As shown in Figure 3 of the original paper, Sentistream is capable of rapidly recovering a stable performance level.\n- *Dataset Proportion:* It's worth noting that the labeled data used for semi-supervised learning constitutes only 0.5% of the total dataset, emphasizing its efficiency.\n\n\n\n\n**Q2: Authors may consider using a running example to explain the concept draft of sentiment stream. The experiments use three different dataset to simulate the data stream and sentiment draft. However, an example of concept draft in a single source can strengthen the need of the application.**\n\nWe are grateful for the reviewer's suggestion to include a running example to elucidate the concept of sentiment stream drift. We concur that a single-source example could offer a more focused illustration. Nevertheless, we hope to highlight that:\n\n- *Dataset Limitations:* The use of multiple datasets (SST, LMRD, Yelp) was necessitated by the absence of single, large-timespan datasets that adequately represent concept drift. We have acknowledged this limitation in our manuscript.\n- *Example of Mixed Streams:* We provide an example comparing the transition from the LMRD to the SST dataset to highlight the complexities of mixed sentiment streams. This serves to emphasize the challenges in handling concept drift across diverse domains.\n- *Future Work:* To directly address your suggestion, we plan to include an experiment using the Sentiment140 dataset (contains 1.6 million tweets,  from 6 Apr 2009 to 25 Jun 2009, with each day representing a distinct time period) and Amazon-Fashion dataset (contains chronological customer reviews related to fashion products from 2010 to 2018, organized into quarterly periods). Preliminary results indicate that our approach outperforms existing methods in terms of accuracy, F1 score, and AUC, while maintaining low latency and high throughput in the below table. Experimental results on a single data stream over a long time span demonstrate the effectiveness of our approach, outperforming other methods.\n\n**Sentiment 140 result**\n\nTrain dataset: First-day data\n\nTest dataset: Other date data\n\n| Method            | Latency(ms) | Throughput(tuples/s) | ACC   | F1    | AUC   |\n|-------------------|-------------|----------------------|-------|-------|-------|\n| Self-learning     | 3.573       | 212                  | 0.637 | 0.630 | 0.662 |\n| Supervised(Bert)  | 36.898      | 241                  | 0.599 | 0.666 | 0.620 |\n| Us module              | -           | -                    | 0.677 | 0.664 | 0.664 |\n| Ss module             | -           | -                    | 0.671 | **0.679** | 0.715 |\n| **Sentistream**       | **0.208**       | **4674**                 | **0.678** | 0.670 | **0.722** |\n\n\n\n\n\n**Amazon (Fashion) result**\n\nTrain dataset: First quarter  data\n\nTest dataset: Other quarter data\n\n| Method            | Latency(ms) | Throughput(tuples/s) | ACC   | F1    | AUC   |\n|-------------------|-------------|----------------------|-------|-------|-------|\n| Self-learning     | 2.201       | 452                  | 0.780 | 0.877 | 0.521 |\n| Supervised(Bert)  | 46.056     | 226                  | 0.760 | 0.827 | 0.500 |\n| Us module                | -           | -                    | 0.830 | 0.892 | 0.872 |\n| Ss module              | -           | -                    | 0.829 | 0.869 | 0.923 |\n| **Sentistream**       | **0.279**       | **3511**                 | **0.855** | **0.900** | **0.935** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe paper is addressing an interesting research problem about the data draft in sentiment analysis. It is of the interest of the general audience of the conference.\n2.\tThe rationale and the design of the proposed framework is elaborated in detail. \n3.\tEmpirical results show that the proposed method outperforms other baselines.\n", "reasons_to_reject": "1.\tEmpirical results show that the unsupervised module is the major source of improvement in the performance. The integration of the semi-supervised module can only slightly improve, or even degrade the overall performance. It cannot justify the need of the semi-supervised module in the framework. In particularly, without the semi-supervised module, the framework does not need any training examples. \n2.\tFollowing this, the proposed framework employs lightweight language model in the unsupervised module. If a more sophisticated language model is used, it is unclear if the semi-supervised module is still needed.\n3.\tAuthors may consider using a running example to explain the concept draft of sentiment stream. The experiments use three different dataset to simulate the data stream and sentiment draft. However, an example of concept draft in a single source can strengthen the need of the application.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a co-training method for sentiment analysis on stream data. The proposed framework consists of three modules. The unsupervised module aims at updating the pre-trained language model, which will shift over time. The goal of the semi-supervised module is to update the model based on the pseudo-labels obtained. The stream merge module is designed to combine the prediction results from the unsupervised module and semi-supervised module. Experiments on publicly available datasets were conducted. Comparison with existing baselines shows the proposed method achieves promising results.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gPoLxniijS", "reviewer": "Reviewer_o7C8", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Authors' Response Letter** to the Reviewers of the Manuscript:     \n\nSentiStream: A Co-Training Framework for Adaptive Online Sentiment Analysis in Evolving Data Streams\n\nThanks again for the detailed comments and constructive feedback. Below we discuss how we have addressed each comment as reflected in the revised manuscript. To ease your review, your question will be **bolded**.\n\n\n**Q1: In the experimental results, they only compare with some standard baseline and not with the state-of-the-art in sentiment analysis on data streams.**\n\nWe appreciate the reviewer's observation regarding the comparison with state-of-the-art methods. We would like to clarify the following:\n\n+ *Lack of Universal SOTA:* It is important to note that there is currently no universally accepted state-of-the-art method for sentiment analysis on data streams, making direct comparisons challenging.\n+ *Selected Advanced Methods:* Despite this, we have included comparisons with advanced methods in the field. Specifically, we have chosen CL-WSAT[1] for continual learning in text classification tasks, and a self-supervised method[2] applied to large-scale Twitter data streams. In our paper, these are respectively referred to as \"weakly-supervised\" and \"self-supervised.\"\n\nBy including these advanced methods, we aim to provide a more comprehensive evaluation of our approach in the context of existing research.\n\n\n**Q2: In the way sentiment polarities are approximated (employing sentiment lexicons and computing mean of positive vs mean of negative words), they miss the context of the review (e.g. a review text of the form \"Even though the restaurant was expensive and the orders were late, I enjoyed the experience\" would be classified as negative, even though it is positive).**\n\nWe appreciate the reviewer's concern regarding the approximation of sentiment polarities and the potential for missing contextual nuances. We would like to clarify the following points:\n\n+ *Scope of Analysis:* Our paper focuses on general sentiment analysis. Targeted sentiment analysis or stance detection, which would require different methodologies, is beyond the scope of this work.\n+ *Contextual Understanding:* While we employ Word2Vec, a non-contextual word embedding method, our approach also integrates Hierarchical Attention Networks (HAN) in the semi-supervised module. HAN is designed to capture the context and nuances within the review text effectively.\n+ *Trade-offs:* We acknowledge that using more advanced contextual embeddings like BERT could provide richer contextual information. However, in scenarios requiring low latency, such embeddings may not be practical. Our approach aims to balance accuracy and real-time usability. In particular, as shown in Table 1, the BERT model’s latency is nearly 100 times that of SentiStream. \n\n| Method             | Latency(ms) | Throughput(tuples/s) |\n|--------------------|-------------|------------|\n| Supervised (BERT)  | 53.17       | 162        |\n| SentiStream        | **0.67**        | **1471**       |\n\n\n+ *Real-time Feasibility:* By combining Word2Vec and HAN, we strive to offer meaningful sentiment predictions with manageable latency, making our approach suitable for online stream environments.\n\n\n\n[1]CL-WSTC: Continual Learning for Weakly Supervised Text Classification on the Internet\n\n[2]Large-scale sentiment learning with limited labels", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors research an interesting topic, that of sentiment analysis of data streams. \n2. The proposed architecture tries to address key issues of stream data, such concept drift and system latency.", "reasons_to_reject": "1. In the experimental results, they only compare with some standard baseline and not with the state-of-the-art in sentiment analysis on data streams\n2. In the way sentiment polarities are approximated (employing sentiment lexicons and computing mean of positive vs mean of negative words), they miss the context of the review (eg a review text of the form \"Even though the restaurant was expensive and the orders were late, I enjoyed the experience\" would be classified as negative, even though it is positive).", "questions_to_authors": "", "paper_topic_and_main_contributions": "This submission introduces SentiStream, a sentiment analysis module for streaming data. The authors present the components of their architecture in detail and proceed in evaluating it on three datasets against four other baselines, reporting on their findings.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. \"Continual Training\" -> \"Continuous Training\"\n2. Even though not entirely wrong, please use LLM (Large Language Models) instead of PLM (Pre-trained Language Models)\n3. All 3 algorithms should be described in more detail (number steps and briefly explain what each step does)."}}, {"review_id": "YrEdPCa4Dv", "reviewer": "Reviewer_55iG", "summary": "", "questions": "- [resolved if added to the paper] ~~L#380: what are the \"specific classifying rules\"?~~\n- resolved if fixed in the paper~~Figure2: is the caption describe the figure correctly?~~", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Authors' Response Letter** to the Reviewers of the Manuscript:     \n\nSentiStream: A Co-Training Framework for Adaptive Online Sentiment Analysis in Evolving Data Streams\n\nThanks again for the detailed comments and constructive feedback. Below we discuss how we have addressed each comment as reflected in the revised manuscript. To ease your review, your question will be **bolded**.\n\n\n**Q1: The main dimension of the research, that is time, is not explained. Simply, how are the reviews ordered? What is the time interval? What is the time span of each dataset? The logic behind mixing the dataset is confusing. How such mixture produce topic drift?**\n\nWe appreciate the reviewer's inquiry into the time dimension and the rationale behind mixing datasets in our research. We would like to clarify the following:\n\n+ *Objective of Dataset Mixing:* The mixing of datasets aims to emulate the complexities of data drift in real-world scenarios. While the datasets are not chronologically ordered, their sequential arrangement captures evolving sentiment expression and language patterns.\n+ *Time Periods and Concepts:* Each dataset represents distinct time periods and explores different concepts sequentially. This setup allows us to observe how sentiment expression evolves and adapts to changing contexts.\n+ *Examples of Shifts:*\n    + *LMRD & SST:* The transition from LMRD to SST demonstrates a clear evolution in language styles and sentiment lexicons, highlighting the influence of context and domain.\n    + *Yelp & LMRD/SST:* The shift between Yelp and LMRD/SST datasets exemplifies how language and sentiment expressions change across different domains.\n\n+ *Additional Experiment:* To further validate our model's performance on real-world datasets, we have included an experiment using Sentiment140 dataset (contains 1.6 million tweets, from 6 Apr 2009 to 25 Jun 2009, with each day representing a distinct time period ) and Amazon-Fashion dataset (contains chronological customer reviews related to fashion products from 2010 to 2018, organized into quarterly periods). Experimental results on a single data stream over a long time span demonstrate the effectiveness of our approach, outperforming other methods.\n\n\n**Sentiment 140 result**\n\nTrain dataset: First-day data\n\nTest dataset: Other date data\n\n| Method            | Latency(ms) | Throughput(tuples/s) | ACC   | F1    | AUC   |\n|-------------------|-------------|----------------------|-------|-------|-------|\n| Self-learning     | 3.573       | 212                  | 0.637 | 0.630 | 0.662 |\n| Supervised(Bert)  | 36.898      | 241                  | 0.599 | 0.666 | 0.620 |\n| Us module                | -           | -                    | 0.677 | 0.664 | 0.664 |\n| Ss module             | -           | -                    | 0.671 | **0.679** | 0.715 |\n| **Sentistream**       | **0.208**       | **4674**                 | **0.678** | 0.670 | **0.722** |\n\n\n\n**Amazon (Fashion) result**\n\nTrain dataset:  First quarter data\n\nTest dataset:  Other quarter data\n\n\n| Method            | Latency(ms) | Throughput(tuples/s) | ACC   | F1    | AUC   |\n|-------------------|-------------|----------------------|-------|-------|-------|\n| Self-learning     | 2.201       | 452                  | 0.780 | 0.877 | 0.521 |\n| Supervised(Bert)  | 46.056     | 226                  | 0.760 | 0.827 | 0.500 |\n| Us module              | -           | -                    | 0.830 | 0.892 | 0.872 |\n| Ss module             | -           | -                    | 0.829 | 0.869 | 0.923 |\n| **Sentistream**      | **0.279**       | **3511**                 | **0.855** | **0.900** | **0.935** |\n\n**Q2: Not sure I understood the \"throughput\" metric, it's relation with the work, and how it is gauging the efficacy of the proposed model.**\n\nWe appreciate the reviewer's query regarding the \"throughput\" metric. We would like to clarify the following:\n\n+ *Definition:* Throughput quantifies the rate at which our model processes incoming data streams, serving as a measure of analysis speed.\n+ *Relevance:* In real-world applications such as online sentiment analysis, where data streams are continuous, rapid analysis is crucial. Thus, throughput is a pivotal metric.\n+ *Efficacy Gauge:* By comparing our model's throughput with that of baseline models, we aim to demonstrate its superior capability to handle high data volumes in real-time.\nThroughput, therefore, provides a practical lens through which to assess the real-time applicability and efficacy of our proposed model.\n\n**Q3: No competitive baseline is used. I understand that there might not many works in temporal sentiment analysis, but there are many similar research directions like in temporal recommender systems that could be adopted to benchmark the proposed method against them.**\n\nWe appreciate the reviewer's suggestion to consider temporal recommender systems as potential baselines. However, we would like to clarify the following:\n\n+ *Domain-Specific Challenges:* While both temporal sentiment analysis and temporal recommender systems involve time-based aspects, the challenges they address are distinct. Our focus is on real-time sentiment classification in dynamic text streams, which differs significantly in granularity, labeling processes, and model complexities from temporal recommender systems.\n+ *Choice of Baselines:* Our choice of standard baselines was deliberate, aiming to strike a balance between accuracy and real-time processing speed. Models like LLaMA and ChatGpt, although potentially more accurate, come with computational complexities that may not be suitable for real-time applications.\n+ *Competitive Basis:* The selected baselines, such as self-supervised and weakly-supervised methods, represent state-of-the-art solutions that offer a strong competitive basis for our comprehensive comparisons. They effectively address the challenges of concept drift and large-scale sentiment analysis with limited labeled data.\n\nGiven these considerations, we believe our choice of baselines is aligned with the specific focus and challenges of our research. We will continue to explore ways to enrich our evaluation methodology while maintaining its applicability to our domain.\n\n**Q4: Accuracy is a legit metric in binary classification but in balance datasets. I'm not sure the datasets in the paper's testbed are balanced. There are well-established metrics like auc that could be used to add more reliability for the results.**\n\nWe appreciate the reviewer's suggestion to consider alternative metrics for more reliable results, especially in the context of potentially imbalanced datasets. \n+ *Additional Metrics:* In response to your suggestion, we have included supplementary experiments focusing on the AUC metric, as shown in the table below. The experimental demonstration illustrates that Sentistream outperforms other methods in  AUC metrics.\n\n| Method            | AUC    |\n|-------------------|--------|\n| random            | 50.19% |\n| Self-supervised   | 47.66% |\n| Supervised (W2V)  | 73.44% |\n| Supervised (BERT) | 78.99% |\n| Us Nodule      | 82.84% |\n| Ss Module         | 82.20% |\n| Sentistream       | **84.97%** |\n\n+ *F1 Score:* We reported the F1 scores in Table 2 of the original paper. The F1 scores are well suited for evaluating the performance of unbalanced datasets.\n+ *Dataset Distributions:* Our method demonstrates superior performance in both balanced and imbalanced datasets compared to other approaches. Detailed dataset distributions have been provided to offer context for the evaluation metrics used. \n\n| Dataset           | Negative | Positive |\n|-------------------|----------|----------|\n| Yelp              | 40227    | 39773    |\n| LMRD              | 24698    | 24884    |\n| SST-2             | 30076    | 37779    |\n| Sentiment140      | 800000   | 800000   |\n| Amazon (Fashion)  | 170924   | 610225   |\n\n\n**Q5: L#380: what are the \"specific classifying rules\"?**\n\nThe term \"specific classifying rules\" pertains to the situation in which the Yelp Review dataset, originally structured with four levels of sentiment polarity (ranging from 1 to 4), was subsequently converted into a two-class classification dataset (1,2 → positive; 3,4 → negative) in our paper. In contrast, LMRD and SST-2 datasets is two-classifications dataset, distinguishing between positive and negative sentiments.\n\n\n**Q6: Figure2: is the caption describe the figure correctly?**\n\n\nThe original caption was not accurate, the correct caption is: Data stream (Yelp → LMRD → SST-2).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ Interesting research direction\n+ Well-motivated and well-structured", "reasons_to_reject": "- The main concern is about the evaluation methodology, which is not sound and includes unclear parts, to my understanding. Specifically:\n-- The main dimension of the research, that is time, is not explained. Simply, how are the reviews ordered? What is the time interval? What is the time span of each dataset? ...\n-- [resolved if added to the paper] ~~Not sure I understood the \"throughput\" metric, it's relation with the work, and how it is gauging the efficacy of the proposed model.~~\n-- The logic behind mixing the dataset is confusing. How such mixture produce topic drift? \n-- No competitive baseline is used. I understand that there might not many works in temporal sentiment analysis, but there are many similar research directions like in temporal recommender systems that could be adopted to benchmark the proposed method against them\n\n-- [resolved] ~~Accuracy is a legit metric in binary classification but in balance datasets. I'm not sure the datasets in the paper's testbed are balanced. There are well-established metrics like auc that could be used to add more reliability for the results.~~", "questions_to_authors": "- [resolved if added to the paper] ~~L#380: what are the \"specific classifying rules\"?~~\n- resolved if fixed in the paper~~Figure2: is the caption describe the figure correctly?~~", "paper_topic_and_main_contributions": "The paper proposes a streaming strategy for sentiment analysis of online reviews. The proposed model is simple yet seemingly effective. It is a self-correcting pipeline of a supervised and a lexicon-based unsupervised classifier that incrementally updates/corrects itself as the reviews are fed into the model in a temporally ordered stream.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "n/a"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- not sure the tilde char (~) is a proper choice for showing a stream. I would suggest a list [x1,x2,...xn]\n- \"using 0.5% of the entire dataset\" ==> \"50%\"?\n- \"simulating real-world data drift.=> \"topic drift\" or \"semantic drift\""}}]

Author rebuttal:

Meta-review: The problem statement, solution, methods, and experiments are clearly explained. The manuscript is well-written and it is very easy to follow. The research topic is interesting and although the individual methods are not novel, the way the methods are combined together, the overall solution is interesting. 

However, one of the major contributions of the work is to build a model that can be adapted to the dynamic of the data, however, the dataset they used does not completely capture the capabilities of the framework they are proposing. The datasets are collected in different years and they have different genres certainly, they are different in nature, but one cannot compare the nature of a static dataset in a specific era of time with the nature of the data stream. 

In addition, another concern is the way sentiment words are added to the seed words with positive and negative polarity. Sentiment is such a subjective phenomenon and it is very hard to add words based on a threshold since words in different contexts could mean differently. This matter is also mentioned by one of the reviewers.

Lastly, all the baseline and other methods that are compared with this framework are static and that is not a fair comparison because these models are static.
Decision: Accept-Main