Repository: praw-dev/praw

Issue:
Hard-coded limit in stream_generator() function.
## Issue Description

When I'm create a subreddit read stream, I cannot configure how many items from the past to take. The value 100 is hard-coded in the function stream_generator(). As a result, if I want to start reading comments in subreddit from the current time, I still get up to 100 previous ones and I need to filter them. In my opinion, this value should be passed in the kwargs of function stream_generator().


## System Information

    PRAW Version: 5.4.0
  Python Version: 3.6
Operating System: Debian 9



Discussion:
@savao what end result would you actually like?
@bboe  I would like the stream to start not with the previous comments, but with the current one.
@savao I think that's doable. How does this sound, PRAW would simply throw away the responses from the first request which contains only old items?
I want to point out that, per previous discussion with @bboe and my own experience, it is sometimes possible for a stream to occasionally yield old items, even after it has been running for a while. Therefore, it may be necessary to filter regardless.
@bboe I do this in my code, but I don't understand, why it is needed? why this parameter is hardcoded and I can't change it. Except to make a fork of the library.
@leviroth I calculate the current time before starting the thread and check the comment creation time before processing, if it is less than the start time, then "continue"
Yes, I do the same thing in one of my scripts. What I'm saying is that, even if `stream_generator` had an option not to yield past items on startup, it might do so accidentally anyway, just because of some quirk of the API. So, it might still be necessary to do the filtering that you describe.
> why this parameter is hardcoded and I can't change it

There is no hardcoded parameter. PRAW simply requests all items that it can. For comment streams, all 100 comments may have been made in the last second, in which case they are relevant, whereas, for some smaller subreddits, it's possible all the items are from more than a year ago.

I think the simplest solution would be to add a `skip_existing` parameter (defaults to `False`), which will simply ignore all items discovered in the first request. Note that "old" items may still be returned, as always is the case, if an old auto-removed submission/comment gets approved. The distinction here is that they will be new to the stream (unless the authenticated user is a moderator of the subreddit the item belongs to).

Resolving patch:
diff --git a/praw/models/util.py b/praw/models/util.py
--- a/praw/models/util.py
+++ b/praw/models/util.py
@@ -76,7 +76,7 @@ def permissions_string(permissions, known_permissions):
     return ','.join(to_set)
 
 
-def stream_generator(function, pause_after=None):
+def stream_generator(function, pause_after=None, skip_existing=False):
     """Yield new items from ListingGenerators and ``None`` when paused.
 
     :param function: A callable that returns a ListingGenerator, e.g.
@@ -90,6 +90,10 @@ def stream_generator(function, pause_after=None):
         ``0`` yields ``None`` after every response resulting in no new items,
         and a value of ``None`` never introduces a pause (default: None).
 
+    :param skip_existing: When True does not yield any results from the first
+        request thereby skipping any items that existed in the stream prior to
+        starting the stream (default: False).
+
     .. note:: This function internally uses an exponential delay with jitter
        between subsequent responses that contain no new results, up to a
        maximum delay of just over a 16 seconds. In practice that means that the
@@ -159,8 +163,10 @@ def stream_generator(function, pause_after=None):
             found = True
             seen_fullnames.add(item.fullname)
             newest_fullname = item.fullname
-            yield item
+            if not skip_existing:
+                yield item
         before_fullname = newest_fullname
+        skip_existing = False
         if valid_pause_after and pause_after < 0:
             yield None
         elif found:
diff --git a/praw/objector.py b/praw/objector.py
--- a/praw/objector.py
+++ b/praw/objector.py
@@ -42,9 +42,12 @@ def kind(self, instance):
         :param instance: An instance of a subclass of RedditBase.
 
         """
+        retval = None
         for key in self.parsers:
             if isinstance(instance, self.parsers[key]):
-                return key
+                retval = key
+                break
+        return retval
 
     def _objectify_dict(self, data):
         """Create RedditBase objects from dicts.
@@ -100,7 +103,7 @@ def objectify(self, data):
         """
         # pylint: disable=too-many-return-statements
         if data is None:  # 204 no content
-            return
+            return None
         if isinstance(data, list):
             return [self.objectify(item) for item in data]
         if 'kind' in data and data['kind'] in self.parsers:


Test patch:
diff --git a/tests/integration/models/reddit/test_subreddit.py b/tests/integration/models/reddit/test_subreddit.py
--- a/tests/integration/models/reddit/test_subreddit.py
+++ b/tests/integration/models/reddit/test_subreddit.py
@@ -7,7 +7,7 @@
                          ModmailConversation, ModmailMessage, Redditor,
                          Submission, Subreddit, SubredditMessage, Stylesheet,
                          WikiPage)
-from prawcore import Forbidden, NotFound, TooLarge
+from prawcore import Forbidden, NotFound, RequestException, TooLarge
 import mock
 import pytest
 
@@ -971,6 +971,21 @@ def test_comments__with_pause(self, _):
             assert comment_count == 17
             assert pause_count == 2
 
+    @mock.patch('time.sleep', return_value=None)
+    def test_comments__with_skip_existing(self, _):
+        with self.recorder.use_cassette('TestSubredditStreams.comments'):
+            generator = self.reddit.subreddit('all').stream.comments(
+                skip_existing=True)
+            count = 0
+            try:
+                for comment in generator:
+                    count += 1
+            except RequestException:
+                pass
+            # This test uses the same cassette as test_comments which shows
+            # that there are at least 400 comments in the stream.
+            assert count < 400
+
     def test_submissions(self):
         with self.recorder.use_cassette('TestSubredditStreams.submissions'):
             generator = self.reddit.subreddit('all').stream.submissions()
@@ -989,6 +1004,20 @@ def test_submissions__with_pause(self, _):
                 submission = next(generator)
             assert submission_count == 100
 
+    @mock.patch('time.sleep', return_value=None)
+    def test_submissions__with_pause_and_skip_after(self, _):
+        with self.recorder.use_cassette('TestSubredditStreams.submissions'):
+            generator = self.reddit.subreddit('all').stream.submissions(
+                pause_after=-1, skip_existing=True)
+            submission = next(generator)
+            assert submission is None  # The first thing yielded should be None
+            submission_count = 0
+            submission = next(generator)
+            while submission is not None:
+                submission_count += 1
+                submission = next(generator)
+            assert submission_count < 100
+
 
 class TestSubredditStylesheet(IntegrationTest):
     @staticmethod