Information Filtering Method for Twitter
Streaming Data Using
Human-in-the-Loop Machine Learning
Yu Suzuki(B) and Satoshi Nakamura
Nara Institute of Science and Technology,
8916-5 Takayama, Ikoma, Nara 6300192, Japanysuzuki@is.naist.jp

Abstract. There are a massive amount of texts on social media. However, only a small portion of these texts is informative for a speciﬁc
purpose. If we accurately ﬁlter the texts in the streams, we can obtain
useful information in real time. In a keyword-based approach, ﬁlters areconstructed using keywords, but selecting the appropriate keywords toinclude is often diﬃcult. In this work, we propose a method for ﬁltering texts that are related to speciﬁc topics using both crowdsourcing
and machine learning based text classiﬁcation method. In our approach,we construct a text classiﬁer using FastText and then annotate whether
the tweets are related to the topics using crowdsourcing. In this step, we
consider two strategies, optimistic and pessimistic approach, for selecting
tweets which should be assessed. Then, we reconstruct the text classiﬁer
using the annotated texts and classify them again. We assume that if we
continue instigating this loop, the accuracy of the classiﬁer will improve,and we will obtain useful information without having to specify keywords. Experimental results demonstrated that our proposed system is
eﬀective for ﬁltering social media streams. Moreover, we conﬁrmed thatthe pessimistic approach is better than the optimistic approach.

1

Introduction

A massive amount of texts are on social network services, and much information about real situations can be gleaned from these texts. There have been
many studies on how to extract the necessary information from these data [1].
Information ﬁltering [2] is a process for retrieving texts from text streaming,and the keyword-based method is often used for this. However, it is diﬃcult to
set appropriate keywords that correspond to the information needed, because of
two reasons; information need are generally vague and sentences on social networks are always broken. Crowdsourcing is one possible solution to solve theseissues. However, assessing all tweet streaming data by crowdsourcing is unrealistic, because there are a large amount of tweets, then we should pay many wages
to the workers for assessing all tweets. To reduce the amount of crowdsourcing
tasks, we used machine learning.
c Springer Nature Switzerland AG 2018

S. Hartmann et al. (Eds.): DEXA 2018, LNCS 11030, pp. 167–175, 2018.
https://doi.org/10.1007/978-3-319-98812-2_13

168

Y. Suzuki and S. Nakamura

In our research, we have developed a system that collects a small numberof texts relevant to subjective queries from among a large text stream by using
machine learning and crowdsourcing. Our objective is to extract tweets related to
topics from text streaming. The topics we assume are general, and the extracted
tweets are worthwhile for many users (i.e., not personalized topics).
To achieve our objective, we combine two techniques; crowdsourcing andmachine learning, which is called Human-in-the-Loop. In this research, we investigated a problem that occurs when active learning is performed on information
ﬁltering with regard to the tweets presented to the workers. Information ﬁltering
was performed using machine learning and crowdsourcing so as to determine theaccuracy and the cost involved in obtaining relevant tweets.

2

Related WorkIn social network services, numerous useful texts are posted, such as those related
to the spread of inﬂuenza or the occurrence of an accident. Several systems havebeen proposed to capture these incidents quickly [3] and thereby to utilize social
media services as a kind of social sensor. Many information ﬁltering methods
to glean useful information from streaming data have been proposed, and manyof them are used in systems generated for personalization [4] in what is called“personalized information ﬁltering.” These techniques are based on information retrieval and are not appropriate for short texts such as tweets. Therefore,bag-of-words features along with domain-speciﬁc knowledge [5], the relationshipbetween users [6], and user behaviors such as re-tweeting [7] are used as features
to ﬁlter tweets.
Relevance feedback is important for improving the accuracy of information
ﬁltering. Rocchio [8] proposed a relevance feedback mechanism on the vector
space model. In this mechanism, since more accuracy is improved as more feedback is given, a method using crowdsourcing for feedback has been proposed.
[9,10]. However, in these methods, it is assumed that the set of documents thatare compatible is suﬃciently large as compared with the whole set, so it was notclear whether it can be used for information ﬁltering. In this research, we clarify whether relevance feedback by crowdsourcing is eﬀective when the relevant
document is extremely few.

3

Information Filtering Method

Our proposed information retrieval system uses a combination of crowdsourcing
and machine learning techniques. An overview is shown in Fig. 1.
3.1

Building Classifier

We use FastText [11,12], an application for word embedding and classiﬁcation,
for classifying tweets. With FastText, we input a training set, a pre-trained
embedding model, and parameters.

Information Filtering Method for Twitter Streaming Data

169

Step 1.
(Fasttext)
TwitterStreaming
API

Step 3.

Step 2.
Annotation(Crowdsourcing)

Filtered Text

Relevance
Feedback
Relevance Feedback

Fig. 1. Overview of the proposed system.

First, we prepare a dataset for constructing an initial classiﬁer. We remove
tweets from the dataset if they include URLs or mention other users. Then, we
prepared a set of tweets that are related to speciﬁc topics using crowdsourcing.
Each tweet is given the label positive or negative. The label positive means thatthe tweets are relevant to the topics, and the label negative means that the tweets
are irrelevant to the topics. At this time, the classiﬁer calculates the certainty.
Next, we extract bag-of-word features from the texts. MeCab1 with the
IPADIC-Neologd dictionary2 was used as a morphological analyzer. There are
typically many newly coined words in the tweets, so we use a dictionary that
includes new words. Then, we extract nouns, verbs, adjectives, and adverbs. We
use a pretrained embedding model3 constructed using the Japanese Wikipedia
copus4 . This enables synonyms to be handled in the same way.
Finally, using the classiﬁer included in FastText implementation [13], weconstruct a classiﬁer of the tweets. We assigned a positive label or a negative
label to the unlabeled tweets.
3.2

Classify Tweets

Next, we classify the tweets obtained from text streaming by using the classiﬁer
described in Sect. 3.1. We classify the new tweets and continue classiﬁcation until
we can look for tweets labeled as positive. The tweets judged as positive by theclassiﬁer perform annotation, which is the next step. Tweets that are judged as
negative are discarded after this step.
3.3

Annotation

As shown in Fig. 2, we manually judged whether or not the tweets classiﬁed as
positive by the classiﬁer were positive. Although accuracy can be improved bymaking judgments (and the more people, the better the accuracy), the accuracyof the ﬁnal classiﬁer improves as the number of judged tweets increases, even if
1

http://taku910.github.io/mecab/.
https://github.com/neologd/mecab-ipadic-neologd.
3
https://qiita.com/Hironsan/items/513b9f93752ecee9e670.
4https://dumps.wikimedia.org/jawiki/20170101/.
2

170

Y. Suzuki and S. Nakamura

the accuracy is low. Each judgment made was considered ﬁnal (i.e., there wasno redundancy by multiple workers or averaging that took place).

Irrelevant

Decision Boundary

Relevant

tweet

Optimistic
ApproachPessimistic
ApproachFig. 2. Intuitive example of our proposed method. (Color ﬁgure online)

Interestingly, the performance of the classiﬁer changed depending on which
text was judged by a worker. Campbell et al. [14] pointed out that there are
diﬀerences in the performance of classiﬁers depending on the sample selection
method. The following two ideas were considered as an optimal approach and a
pessimistic approach.
Strategy 1: Optimistic Approach. When a classiﬁer classiﬁes a tweet, it is a
way to prioritize tweets that are diﬃcult to judge as positive or negative. Lewis
et al. [15] proposed a technique called uncertainty sampling, which Simon et al.
[16] later used in an application using support vector machines. In this method,
tweets near a decision boundary are annotated. The blue part in Fig. 2 shows
the tweets which should be annotated using this approach.
Strategy 2: Pessimistic Approach. In this strategy, a presentation is madeto the worker in descending order of probability of what the classiﬁer judges as
positive. In the information ﬁltering handled in this research, many tweets are
considered to be unrelated, which means that even if the positive probabilityof the classiﬁcation result is high, it is unlikely to be positive. Therefore, thismethod was devised to judge, as accurately as possible, the appropriate tweet.
The red portion in Fig. 2 shows the tweets which should be annotated using thisapproach.
This method is considered suitable for when the classiﬁcation performanceby classiﬁers is not suﬃcient for the unlabeled tweet.

Information Filtering Method for Twitter Streaming Data

171

Table 1. Parameters for FastText.
Parameter

Value

Number of epochs
Size of vectors
Number of buckets
Loss function
Number of negatives sampled
Minimum number of word occurrences
Max length of word n-gram
Learning rate

10,000300
100,000,000
Negative sampling
10
1
1
0.075

Table 2. Experimental setting.
Optimistic Pessimistic
Manually processed tweets 94,597
72
No. of workers

4176,238
72

Evaluation

We used the optimistic and pessimistic approaches (Sect. 3.3) to determine theeﬀectiveness of these strategies using the number of adequate tweets to beobtained (Table 2).
Data. We prepared two groups of tweet data: labeled and unlabeled. As labeled
data, we used all 3,580 manually collected tweets. As unlabeled data, the Twitter Streaming API was used to collect more than 1 million tweets in advance.
To ensure the same conditions when making a comparison of the two strategies, the tweets to be categorized were made identical for both strategies. If theperformance was the same, the same tweet was extracted.
Procedure. Evaluation experiments were carried out as follows.
– Build a classiﬁer using labeled data.
– Arrange unlabeled data in chronological order and classify using a classiﬁer.
Obtain 1,000 positive tweets.
– Classify tweets against positive tweets by crowdsourcing.
– Add judgment result of crowdsourcing to labeled data and return to 1.
First, we gathered tweets by using the optimistic strategy and then collected
tweets again by the pessimistic strategy. The number of workers was the same
in both strategies, but the workers themselves were diﬀerent.

172

Y. Suzuki and S. Nakamura

Number of correct tweets

100

Optimistic
Pessimistic

80
60
40
20
0

0

20000

40000

60000

80000

Number of assessed tweets

Fig. 3. Ratio of positive tweets and correct answer rate.

Results and Discussion. The hyperparameter used for the classiﬁer is shownin Table 1. In a preliminary experiment using initially labeled data, parametersthat can classify positive and negative with high accuracy were obtained by gridsearch.
We obtained 94,598 and 176,238 assessments by using the optimistic and pessimistic approaches, respectively. For comparison, we used all 94,548 assessments
by the optimistic approach and the ﬁrst 94,548 assessments by the pessimistic
approach. The results are shown in Fig. 3. We discovered that the correct tweet
could be collected twice as fast by the pessimistic approach than the optimistic
approach. Speciﬁcally, when the number of the assessed tweets was 40,000, thesystem could collect many correct tweets when it exceeded 85,000.
Figure 4 shows the number of model reconstruction (steps) vs. the ratio of
positive tweets. In this ﬁgure, a point shows how much percentage of new tweets
classiﬁer judged as positive at a step. For example, if the value at step 3 is0.1, the classiﬁer judges the tweets as positive at the ratio of 1 to 10 when theclassiﬁer was rebuilt three times. From this ﬁgure, in the pessimistic approach,many tweets were judged as positive in the ﬁrst three steps, but after step 4
the ratio is lower than the optimistic approach and the ratio is converged toabout 0.02. On the other hand, in the optimistic approach, in the ﬁrst step, theclassiﬁer judge a small number of the positive tweet than that by the pessimistic
approach. However, the ratio of positive tweets does not decrease in subsequent
steps. The same number of positive tweets are selected at each level. Therefore,
decreasing the ratio of positive tweets means that it is possible to judge many
tweets in a short time. As a result, it was found that the pessimistic approach
can handle many tweets as compared with the optimistic approach.

Information Filtering Method for Twitter Streaming Data
0.25

173

Optimistic Approach
Pessimistic Approach

Ratio of positive tweets

0.2

0.15

0.1

0.05

0

0

5

10

15

20

25

30

35

Steps

Fig. 4. The number of steps vs. ratio of positive tweets

5Conclusion

In this paper, we proposed a method for ﬁltering tweet streams using both crowdsourcing and machine learning. In this research, we investigated a problem that
occurs when active learning is performed on information ﬁltering with regard to
the tweets presented to the workers. Information ﬁltering was performed using
machine learning and crowdsourcing so as to determine the accuracy and thecost involved in obtaining relevant tweets.
In the evaluation experiment, we conﬁrmed how many relevant tweets thesystem can collect when combining crowdsourcing and FastText, a machinelearning based classiﬁer. At this time, we compared two strategies, optimistic
and pessimistic approach, to select tweets presented to workers which are usefulfor improving the accuracy of the classiﬁer. As a result, we were able to collectrelevant tweets as much as 50 tweets and 100 tweets for optimistic approach,and pessimistic approach for 4,500 JPY (45 USD), respectively. At this time,
we got an assessment result of around 95,000. We were able to obtain unfavorable tweets with keywords so that we could show the usefulness of the proposed
method.
In future work, we should combine the existing keyword-based approach with
our proposed crowdsourcing and machine learning based approach for constructing more accurate information ﬁltering.
Acknowledgments. The research results have been achieved by “Research and Development on Fundamental and Utilization Technologies for Social Big Data,” the Commissioned Research of National Institute of Information and Communications Technology (NICT), JAPAN.

174

Y. Suzuki and S. Nakamura

References
1. Banko, M., Cafarella, M.J., Soderland, S., Broadhead, M., Etzioni, O.: Open information extraction from the web. In: Proceedings of the 20th International Joint
Conference on Artiﬁcal Intelligence, IJCAI 2007, pp. 2670–2676, San Francisco,
CA, USA, Morgan Kaufmann Publishers Inc. (2007)
2. Belkin, N.J., Croft, W.B.: Information ﬁltering and information retrieval: two sidesof the same coin? Commun. ACM 35(12), 29–38 (1992)
3. Abel, F., Hauﬀ, C., Houben, G.J., Stronkman, R., Tao, K.: Twitcident: ﬁghting ﬁrewith information from social web streams. In: Proceedings of the 21st International
Conference on World Wide Web, WWW 2012 Companion, pp. 305–308, New York.
ACM (2012)
4. Shardanand, U., Maes, P.: Social information ﬁltering: algorithms for automating &ldquo;word of mouth&rdquo;. In: Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems, CHI 1995, pp. 210–217, New York. ACM
Press/Addison-Wesley Publishing Co. (1995)
5. Sriram, B., Fuhry, D., Demir, E., Ferhatosmanoglu, H., Demirbas, M.: Short textclassiﬁcation in twitter to improve information ﬁltering. In: Proceedings of the 33rd
International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR 2010, pp. 841–842, New York. ACM (2010)
6. Hannon, J., Bennett, M., Smyth, B.: Recommending twitter users to follow using
content and collaborative ﬁltering approaches. In: Proceedings of the Fourth ACM
Conference on Recommender Systems, RecSys 2010, pp. 199–206, New York. ACM(2010)
7. Uysal, I., Croft, W.B.: User oriented tweet ranking: a ﬁltering approach to
microblogs. In: Proceedings of the 20th ACM International Conference on Information and Knowledge Management, CIKM 2011, pp. 2261–2264, New York. ACM(2011)
8. Rocchio, J.J.: Relevance feedback in information retrieval. In: Salton, G. (ed.) The
Smart Retrieval System: Experiments in Automatic Document Processing, pp.
313–323. Prentice Hall (1971)
9. Grady, C., Lease, M.: Crowdsourcing document relevance assessment with mechanical turk. In: Proceedings of the NAACL HLT 2010 Workshop on Creating Speechand Language Data with Amazon’s Mechanical Turk, CSLDAMT 2010, pp. 172–
179, Stroudsburg, PA, USA. Association for Computational Linguistics (2010)
10. Alonso, O., Baeza-Yates, R.: Design and implementation of relevance assessments using crowdsourcing. In: Clough, P., et al. (eds.) Advances in Information
Retrieval, pp. 153–164. Springer, Heidelberg (2011). https://doi.org/10.1007/9783-642-20161-5 16
11. Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors withsubword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017)12. Joulin, A., Grave, E., Bojanowski, P., Mikolov, T.: Bag of tricks for eﬃcient textclassiﬁcation. In: Proceedings of the 15th Conference of the European Chapterof the Association for Computational Linguistics: Volume 2, Short Papers, pp.
427–431. Association for Computational Linguistics (2017)13. Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., Mikolov, T.: Fasttext.zip: compressing text classiﬁcation models, December 2016
14. Campbell, C., Cristianini, N., Smola, A.J.: Query learning with large margin classiﬁers. In: Proceedings of the Seventeenth International Conference on Machine
Learning, ICML 2000, pp. 111–118, San Francisco, CA, USA, Morgan Kaufmann
Publishers Inc. (2000)

Information Filtering Method for Twitter Streaming Data

175

15. Lewis, D.D., Gale, W.A.: A sequential algorithm for training text classiﬁers. In:
Croft, B.W., van Rijsbergen, C.J. (eds.) SIGIR 1994, pp. 3–12. Springer, New York(1994). https://doi.org/10.1007/978-1-4471-2099-5 1
16. Tong, S., Koller, D.: Support vector machine active learning with applications totext classiﬁcation. J. Mach. Learn. Res. 2, 45–66 (2002)