Project: springxd
Issue XD-2944: twittersearch stream returns duplicate tweets
The twittersearch source module picks up the same tweet in successive iterations of the REST request. The reason for this is that the since_id value being set at the end of each iteration is the last item in the list of tweets, but not the latest value of tweet id. Steps to reproduce: Created a stream using below definition stream create --name twittersearchspring --definition "twittersearch --consumerKey= --consumerSecret= --query='spring' | tweet-transformer | file" --deploy tweet-transformer referred here is used from the spring-xd-samples repo and is logging the tweet ID being transformed https://github.com/spring-projects/spring-xd-samples/tree/master/tweet-transformer-processor Below is the log __________ 2015-04-14 15:47:22,154 1.1.0.RELEASE INFO DeploymentSupervisor-0 server.StreamDeploymentListener - Deployment status for stream 'twittersearchspring': DeploymentStatus{state=deployed} 2015-04-14 15:47:22,158 1.1.0.RELEASE INFO DeploymentSupervisor-0 server.StreamDeploymentListener - Stream Stream{name='twittersearchspring'} deployment attempt complete 2015-04-14 15:47:24,301 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884756488032256 2015-04-14 15:47:24,312 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884752591544320 2015-04-14 15:47:24,315 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884752536911873 2015-04-14 15:47:24,318 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884751219986432 2015-04-14 15:47:24,320 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884750620258305 2015-04-14 15:47:24,325 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884749974147072 2015-04-14 15:47:24,340 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884743338917888 2015-04-14 15:47:24,342 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884741493272577 2015-04-14 15:47:24,343 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884739387719680 2015-04-14 15:47:24,344 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884737307549696 2015-04-14 15:47:24,346 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884734723702784 2015-04-14 15:47:24,348 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884730059771905 2015-04-14 15:47:24,356 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884729225125888 2015-04-14 15:47:24,358 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884725802405888 2015-04-14 15:47:24,359 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884724938481665 2015-04-14 15:47:26,465 1.1.0.RELEASE DEBUG twitterSource-1-1 twitter.TwitterSearchChannelAdapter - Search uri:https://api.twitter.com/1.1/search/tweets.json?q=spring&since_id=587884724938481665 2015-04-14 15:47:26,865 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884766608916481 2015-04-14 15:47:26,867 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884766072025089 2015-04-14 15:47:26,869 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884765707116544 2015-04-14 15:47:26,871 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884764394299392 2015-04-14 15:47:26,872 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884761366003712 2015-04-14 15:47:26,878 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884759381966849 2015-04-14 15:47:26,880 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884756488032256 2015-04-14 15:47:26,882 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884752591544320 2015-04-14 15:47:26,884 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884752536911873 2015-04-14 15:47:26,886 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884751219986432 2015-04-14 15:47:26,887 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884750620258305 2015-04-14 15:47:26,889 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884749974147072 2015-04-14 15:47:26,890 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884743338917888 2015-04-14 15:47:26,900 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884741493272577 2015-04-14 15:47:26,903 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884739387719680 2015-04-14 15:47:29,004 1.1.0.RELEASE DEBUG twitterSource-1-1 twitter.TwitterSearchChannelAdapter - Search uri:https://api.twitter.com/1.1/search/tweets.json?q=spring&since_id=587884739387719680 2015-04-14 15:47:29,369 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884782945611776 2015-04-14 15:47:29,371 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884781305663488 2015-04-14 15:47:29,374 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884779506311169 2015-04-14 15:47:29,376 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884775895072768 2015-04-14 15:47:29,377 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884771893739520 2015-04-14 15:47:29,379 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884771717578753 2015-04-14 15:47:29,384 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884769138081792 2015-04-14 15:47:29,386 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884766608916481 2015-04-14 15:47:29,388 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884766072025089 2015-04-14 15:47:29,389 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884765707116544 2015-04-14 15:47:29,390 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884764394299392 2015-04-14 15:47:29,391 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884761366003712 2015-04-14 15:47:29,395 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884759381966849 2015-04-14 15:47:29,399 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884756488032256 2015-04-14 15:47:29,401 1.1.0.RELEASE INFO twitterSource-1-1 transformer.TweetTransformer - Transforming tweet with id : 587884752591544320 2015-04-14 15:47:31,502 1.1.0.RELEASE DEBUG twitterSource-1-1 twitter.TwitterSearchChannelAdapter - Search uri:https://api.twitter.com/1.1/search/tweets.json?q=spring&since_id=587884752591544320 __________ Notice that tweet ID 587884752591544320 is picked up in second iteration as well although it was picked in the first. The issue can be fixed in the doSendLine method of TwitterSearchChannelAdapter.java where this.sinceId is being set to the last value of id. Instead, the statuses map can be sorted on ID, and the highest ID can be set for sinceId.
Story points: 2