Package: psaw
Detailed documentation for PSAW is available at:https://psaw.readthedocs.io/en/latest/InstallationpipinstallpsawDescriptionA minimalist wrapper for searching public reddit comments/submissions via the pushshift.io API.Pushshift is an extremely useful resource, but the API is poorly documented. As such, this API wrapper
is currently designed to make it easy to pass pretty much any search parameter the user wants to try.Although it is not necessarily reflective of the current status of the API, you should
attempt to familiarize yourself with the Pushshift API documentation to better understand
what search arguments are likely to work.API Documentation on githubEndpoints and parameter descriptions/r/pushshiftFeaturesHandles rate limiting and exponential backoff subject to maximum retries and
maximum backoff limits. A minimum rate limit of 1 request per second is used
as a default per consultation with Pushshift’s maintainer,/u/Stuck_in_the_matrix.Handles paging of results. Returns all historical results for a given query by default.Optionally handles incorporation ofprawto fetch objects after getting ids from pushshiftIf not usingpraw, returns results incommentandsubmissionobjects whose
API is similar to the correspondingprawobjects. Additionally, result objects have
an additional.d_attribute that offers dict access to the associated data attributes.Optionally adds acreatedattribute which converts a comment/submission’screated_utctimestamp to the user’s local time. (may raise exceptions for users with certain timezone
settings).Simple interface to pass query arguments to the API. The API is sparsely documented,
so it’s often fruitful to just try an argument and see if it works.Astop_conditionargument to make it simple to stop yielding results given arbitrary user-defined criteriaCommandline interface (CLI) for simplified usage outside of python environment.WARNINGSUsing non-default sort may result in unexpected behavior.Default behavior is to continuously hit the pushshift api. If a query is taking
longer than expected to return results, it’s possible that psaw is pulling more data
than you may want or is caught in some kind of loop.I strongly recommend prototyping queries by printing to stdout to ensure you’re getting the
desired behavior.Demo usage (python)frompsawimportPushshiftAPIapi=PushshiftAPI()Or to use pushshift search to fetch ids and then use praw to fetch objects:importprawfrompsawimportPushshiftAPIr=praw.Reddit(...)api=PushshiftAPI(r)100 most recent submissions# The `search_comments` and `search_submissions` methods return generator objectsgen=api.search_submissions(limit=100)results=list(gen)First 10 submissions to /r/politics in 2017, filtering results to url/author/title/subreddit fields.Thecreated_utcfield will be added automatically (it’s used for paging).importdatetimeasdtstart_epoch=int(dt.datetime(2017,1,1).timestamp())list(api.search_submissions(after=start_epoch,subreddit='politics',filter=['url','author','title','subreddit'],limit=10))Trying a search argument that doesn’t actually workAccording to the pushshift.io API documentation, we should be able to search submissions by url,
but (at the time of this writing) this doesn’t actually work in practice.
The API should still respect thelimitargument and possibly other supported arguments,
but no guarantees. If you find that an argument you have passed is not supported by the API,
best thing is to just remove it from the query and modify your api call to only utilize
supported arguments to mitigate risks from of unexpected behavior.url='http://www.politico.com/story/2017/02/mike-flynn-russia-ties-investigation-235272'url_results=list(api.search_submissions(url=url,limit=500))len(url_results),any(r.url==urlforrinurl_results)# 500, FalseAll AskReddit comments containing the text “OP”Use theqparameter to search text. Omitting thelimitparameter does a full
historical search. Requests are performed in batches of size specified by themax_results_per_requestparameter (default=500). Omitting the “max_reponse_cache”
test in the demo below will return all results. Otherwise, this demo will perform two
API requests returning 500 comments each. Alternatively, the generator can be queried for additional results.gen=api.search_comments(q='OP',subreddit='askreddit')max_response_cache=1000cache=[]forcingen:cache.append(c)# Omit this test to actually return all results. Wouldn't recommend it though: could take a while, but you do you.iflen(cache)>=max_response_cache:break# If you really want to: pick up where we left off to get the rest of the results.ifFalse:forcingen:cache.append(c)Using theaggsargument to summarize search resultsWhen an aggs parameter is provided to a search method, the first result yielded by the generator
will contain the aggs result.api=PushshiftAPI()gen=api.search_comments(author='nasa',aggs='subreddit')next(gen)#  {'subreddit': [#    {'doc_count': 300, 'key': 'IAmA'},#    {'doc_count': 6, 'key': 'space'},#    {'doc_count': 1, 'key': 'ExposurePorn'},#    {'doc_count': 1, 'key': 'Mars'},#    {'doc_count': 1, 'key': 'OldSchoolCool'},#    {'doc_count': 1, 'key': 'news'},#    {'doc_count': 1, 'key': 'pics'},#    {'doc_count': 1, 'key': 'reddit.com'}]}len(list(gen))# 312Using theredditor_subreddit_activityconvenience methodIf you want to profile a redditors activity as in theaggsexample, theredditor_subreddit_activityprovides a simple shorthand for profiling a user by the subreddits
in which they are active, counting comments and submissions separately in a single call,
and returning Counter objects for commenting and posting activity, respectively.api=PushshiftAPI()result=api.redditor_subreddit_activity('nasa')result#{'comment':#   Counter({#      'ExposurePorn': 1,#      'IAmA': 300,#      'Mars': 1,#      'OldSchoolCool': 1,#      'news': 1,#      'pics': 1,#      'reddit.com': 1,#      'space': 6}),# 'submission':#   Counter({#      'IAmA': 3,#      'ISS': 1,#      'Mars': 1,#      'space': 3,#      'u_nasa': 86})}Using thestop_conditionargument to get the most recent submission by a bot accountgen=api.search_submissions(stop_condition=lambdax:'bot'inx.author)forsubmingen:passprint(subm.author)Collecting results in apandas.DataFramefor analysisimportpandasaspddf=pd.DataFrame([thing.d_forthingingen])Special Convenience AttributesConsider the following simple query:gen=api.search_submissions(subreddit='pushshift')thing=next(gen)Special attributes:thing.d_a dict containing all of the data attributes attached to the thing (which otherwise would be accessed via dot notation). One specific convenience this enables is simplifying pushing results into a pandas dataframe (above).api.metadata_The metadata data provided by pushshift (if any) from the most recent successful request. The most useful metadata attributes, IMHO, are:api.metadata_.get('shards')- For checking if any shards are down, which can impact the result cardinality.api.metadata_.get('total_results')- The database-side count of how many total items were found in the query and should be returned after paging through all results. Users have encountered rare edge cases that don’t return all expected results, probably due to more than 500 items sharing the same timestamp in a result range. Seeissue #47for progress resolving this behavior.Demo usage (CLI)For CLI documentation, runpsaw --helpLicensePSAW’s source is provided under theSimplified BSD License.Copyright (c), 2018, David Marx