Package: pmaw
PMAW: Pushshift Multithread API WrapperContentsDescriptionGetting StartedFeaturesMultithreadingRate LimitingCachingPRAW EnrichmentCustom FilteringUnsupported ParametersParametersExamplesCommentsSubmissionsAdvanced ExamplesPRAWCustom FilterCaching ExamplesBenchmarksDeprecated ExamplesDescriptionPMAWis a wrapper for the Pushshift API which uses multithreading to retrieve Reddit comments and submissions. General usage is through thePushshiftAPIclass which provides methods for interacting with differentPushshiftendpoints, please view thePushshift Docsfor more details on the endpoints and accepted parameters. Parameters are provided through keyword arguments when calling the method, some methods will have required parameters. When using a methodPMAWwill complete all the required API calls to complete the query before returning aResponsegenerator object.The following three methods are currently supported:Searching Comments:search_commentsDetailsSearch Submissions:search_submissionsDetailsSearch Submission Comment IDs:search_submission_comment_idsDetailsGetting StartedWhy Multithread?When building large datasets from Reddit submission and comment data it can require thousands of API calls to the Pushshift API. The time it takes for your code to complete pulling all this data is limited by both your network latency and the response time of the Pushshift server, which can vary throughout the day.Current API libraries such as PRAW and PSAW currently run requests sequentially, which can cause thousands of API calls to take many hours to complete. Since API requests are I/O-bound they can benefit from being run asynchronously using multiple threads. Implementing intelligent rate limiting can ensure that we minimize the number of rejected requests, and the time it takes to complete.InstallationPMAWcurrently supportsPython 3.5or later. To install it viapip, run:$pipinstallpmawGeneral UsagefrompmawimportPushshiftAPIapi=PushshiftAPI()View the optional parameters forPushshiftAPIhere.FeaturesMultithreadingThe number of threads to use during multithreading is set with thenum_workersparameter. This is optional and defaults to10, however, you should provide a value as this may not be appropriate for your machine. Increasing the number of threads you use allows you to make more concurrent requests to Pushshift, however, the returns are diminishing as requests are constrained by the rate-limit. The optimal number of threads for requests is between10and20depending on the current response time of the Pushshift server.When selecting the number ofthreadsyou can follow one of the two methodologies:Number of processors on the machine, multiplied by 5Minimum value of 32 and the number of processors plus 4If you are unsure how many processors you have use:os.cpu_count().Rate LimitingMultiple different options are available for rate-limiting your Pushshift API requests, and are defined by two different types, rate-averaging and exponential backoff. If you're unsure on which to use, refer to thebenchmark comparison.Rate-AveragingPMAWby default rate limits using rate-averaging so that the concurrent API requests to the Pushshift server are limited to your provided rate.Providing arate_limitvalue is optional, this defaults to60requests per minute which is the recommended value for interacting with the Pushshift API. Increasing this value above60will increase the number of rejected requests and will increase the burden on the Pushshift server. A maximum recommended value is100requests per minute.Additionally, the rate-limiting behaviour can be constrained by themax_sleepparameter which allows you to select a maximum period of time to sleep between requests.Exponential BackoffExponential backoff can be used by setting thelimit_typeto backoff. Four flavours ofbackoffare available based on the usage of jitter: None, full, equal, and decorr - decorrelated.Exponential backoff is calculated by multiplying thebase_backoffby 2 to the power of the number of failed batches. This allows batches to be spaced out, reducing the resulting rate-limit when requests start to be rejected. However, the threads will still be requesting at nearly the same time, increasing the overall number of required API requests. The exponential backoff sleep values are capped by themax_sleepparameter.Introducing an element of randomness calledjitterallows us to reduce the competition between threads and distribute the API requests across the window, reducing the number of rejected requests.fulljitter selects the length of sleep for a request by randomly sampling from a normal distribution for values between 0 and the capped exponential backoff value.equaljitter selects the length of sleep for a request by adding half the capped exponential backoff value to a random sample from a normal distribution between 0 and half the capped exponential backoff value.decorr- decorrelated jitter is similar tofulljitter but increases the maximum jitter based on the last random value, selecting the length of sleep by the minimum value betweenmax_sleepand a random sample between thebase_backoffand the last sleep value multiplied by 3.CachingMemory SafetyMemory safety allows us to reduce the amount of RAM used when requesting data, and can be enabled by settingmem_safe=Trueon a search method. This feature should be used if a large amount of data is being requested or if the machine in use has a limited amount of RAM.When enabled,PMAWcaches the responses retrieved every 20 batches (approx 20,000 responses with 10 workers) by default, this can be changed by passing a different value forfile_checkpointwhen instantiating thePushshiftAPIobject.When the search is complete, aResponsegenerator object is returned, when iterating through the responses using this generator, responses from the cache will be loaded in 1 cache file at a time.Safe ExitingSafe exiting will ensure that if a search method is interrupted that any unfinished requests and current responses are cached before exiting. If the search method successfully completes, all the responses are also cached. This can be enabled by settingsafe_exit=Trueon a search method.Re-running asearchmethod with the exact same parameters that you have ran before will load previous responses and any unfinished requests from the cache, allowing it to resume if all the required responses have not yet been retrieved. If there are no unfinished requests, the responses from the cache are returned.Abeforevalue is required to load previous responses / requests when using non-id based search, asbeforeis set to the current time when thesearchmethod is called, which would result in a different set of parameters then when you last ran the search despite all other parameters being the same.Similarly to the memory safety feature, aResponsegenerator object is returned. When iterating through the responses using this generator, responses from the cache will be loaded in 1 cache file at a time.PRAW EnrichmentEnrich results with the most recent metadata from Reddit by passing a PRAW Reddit instance when instantiating the PushshiftAPI. Results not found on Reddit will not be enriched or returned.If you don’t already have a client ID and client secret, follow Reddit’sFirst Steps Guideto create them. A user agent is a unique identifier that helps Reddit determine the source of network requests. To use Reddit’s API, you need a unique and descriptive user agent.Custom FilteringA user-defined function can be provided using thefilter_fnparameter for either thesearch_submissionsorsearch_commentsmethod. This function will be used to filter results before they are saved by passing each item to the function and filtering it out if aFalsevalue is returned, saving the value ifTrueis returned. Thelimitparameter does not take into account any results that are filtered out.Unsupported Parametersorder='asc'is unsupported as it can have unexpected resultsuntilandsinceonly support epoch time (float or int)aggsare unsupported, asPMAWis intended to be used for collecting large numbers of submissions or comments. UsePSAWfor aggregation requests.Feature RequestsFor feature requests please open an issue with thefeature requestlabel, this will allow features to be better prioritized for future releasesParametersObjectsPushshiftAPInum_workers(int, optional): Number of workers to use for multithreading, defaults to 10.max_sleep(int, optional): Maximum rate-limit sleep time (in seconds) between requests, defaults to 60s.rate_limit(int, optional): Target number of requests per minute for rate-averaging, defaults to 60 requests per minute.base_backoff(float, optional): Base delay in seconds for exponential backoff, defaults to 0.5sbatch_size(int, optional): Size of batches for multithreading, defaults to number of workers.shards_down_behavior(str, optional): Specifies how PMAW will respond if some shards are down during a query. Options are 'warn' to only emit a warning, 'stop' to throw a RuntimeError, or None to take no action. Defaults to 'warn'.limit_type(str, optional): Type of rate limiting to use, options are 'average' for rate averaging, 'backoff' for exponential backoff. Defaults to 'average'.jitter(str, optional): Jitter to use with backoff, options are None, 'full', 'equal', 'decorr'. Defaults to None.checkpoint(int, optional): Size of interval in batches to print a checkpoint with stats, defaults to 10file_checkpoint(int, optional): Size of interval in batches to cache responses when using mem_safe, defaults to 20praw(praw.Reddit, optional): Used to enrich the Pushshift items retrieved with metadata directly from RedditResponseResponseis a generator object which will return the responses once when iterated over.len(Response)will return the number of responses that were retrieved from Pushshiftload_cache(key, cache_dir=None)returns an instance ofResponsewith the responses loaded with the provided keysearch_submissionsandsearch_commentsmax_ids_per_request(int, optional): Maximum number of ids to use in a single request, defaults to 500, maximum 500.max_results_per_request(int, optional): Maximum number of items to return in a single non-id based request, defaults to 100, maximum 100.mem_safe(boolean, optional): If True, stores responses in cache during operation, defaults to Falsesearch_window(int, optional): Size in days for search window for submissions / comments in non-id based search, defaults to 365safe_exit(boolean, optional): If True, will safely exit if interrupted by storing current responses and requests in the cache. Will also load previous requests / responses if found in cache, defaults to Falsecache_dir(str, optional) - An absolute or relative folder path to cache responses in whenmem_safeorsafe_exitis enabledfilter_fn(function, optional) - A function used for custom filtering the results before saving them. Accepts a single comment or submission parameter and returns False to filter out the item, otherwise returns True.Keyword ArgumentsUnlike the Pushshift API, theuntilandsincekeyword arguments must be in epoch timelimitis the number of submissions/comments to return. If set toNoneor if the setlimitis higher than the number of available submissions/comments for the provided parameters thenlimitwill be set to the amount available.Other accepted parameters are covered in the Pushshift documentation forsubmissionsandcomments.search_submission_comment_idsidsis a required parameter and should be an array of submission ids, a single id can be passed as a stringmax_ids_per_request(int, optional): Maximum number of ids to use in a single request, defaults to 500, maximum 500.mem_safe(boolean, optional): If True, stores responses in cache during operation, defaults to Falsesafe_exit(boolean, optional): If True, will safely exit if interrupted by storing current responses and requests in the cache. Will also load previous requests / responses if found in cache, defaults to Falsecache_dir(str, optional) - An absolute or relative folder path to cache responses in whenmem_safeorsafe_exitis enabledKeyword ArgumentsOther accepted parameters are covered in the PushshiftdocumentationExamplesThe following examples are forpmawversion >= 1.0.0.CommentsSearch CommentsfrompmawimportPushshiftAPIapi=PushshiftAPI()comments=api.search_comments(subreddit="science",limit=1000)comment_list=[commentforcommentincomments]Search Comments by IDsfrompmawimportPushshiftAPIapi=PushshiftAPI()comment_ids=['gjacwx5','gjad2l6','gjadatw','gjadc7w','gjadcwh','gjadgd7','gjadlbc','gjadnoc','gjadog1','gjadphb']comments=api.search_comments(ids=comment_ids)comment_list=[commentforcommentincomments]You can supply a single comment by passing the id as a string or an array with a length of 1 toidsDetailed ExampleSearch Comment IDs by Submission IDfrompmawimportPushshiftAPIapi=PushshiftAPI()post_ids=['kxi2w8','kxi2g1','kxhzrl','kxhyh6','kxhwh0','kxhv53','kxhm7b','kxhm3s','kxhg37','kxhak9']comment_ids=api.search_submission_comment_ids(ids=post_ids)comment_id_list=[c_idforc_idincomment_ids]You can supply a single submission by passing the id as a string or an array with a length of 1 toidsDetailed ExampleSubmissionsSearch SubmissionsfrompmawimportPushshiftAPIapi=PushshiftAPI()posts=api.search_submissions(subreddit="science",limit=1000)post_list=[postforpostinposts]Search Submissions by IDsfrompmawimportPushshiftAPIapi=PushshiftAPI()post_ids=['kxi2w8','kxi2g1','kxhzrl','kxhyh6','kxhwh0','kxhv53','kxhm7b','kxhm3s','kxhg37','kxhak9']posts=api.search_submissions(ids=post_ids)post_list=[postforpostinposts]You can supply a single submission by passing the id as a string or an array with a length of 1 toidsDetailed ExampleAdvanced ExamplesPRAWimportprawfrompmawimportPushshiftAPIreddit=praw.Reddit(client_id='YOUR_CLIENT_ID',client_secret='YOUR_CLIENT_SECRET',user_agent=f'python: PMAW request enrichment (by u/YOUR_USERNAME)')api_praw=PushshiftAPI(praw=reddit)comments=api_praw.search_comments(q="quantum",subreddit="science",limit=100,until=1629990795)Custom FilterThe user defined function must accept a single item (comment / submission) and return either True or False, returning False will filter out the item passed to it.frompmawimportPushshiftAPIapi=PushshiftAPI()deffxn(item):returnitem['score']>2posts=api.search_submissions(ids=post_ids,filter_fn=fxn)Caching ExamplesMemory SafetyIf you are pulling large amounts of data or have a limited amount of RAM, using the memory safety feature will help you avoid an out of memory error from being thrown during data retrieval.frompmawimportPushshiftAPIapi=PushshiftAPI()posts=api.search_submissions(subreddit="science",limit=700000,mem_safe=True)print(f'{len(posts)}posts retrieved from Pushshift')AResponsegenerator object will be returned, and you can load all the responses, including those that have been cached by iterating over the entire generator.# get all responsespost_list=[postforpostinposts]With default settings, responses are cached every 20 batches (approx 20,000 responses with 10 workers), however, with limited memory you can decrease this further.# cache responses every 10 batchesapi=PushshiftAPI(file_checkpoint=10)Safe ExitingIf you expect that your query may be interrupted while its running, settingsafe_exit=Truewill cache responses and unfinished requests before exiting when an interrupt signal is received. Re-running asearchmethod with the exact same parameters that you have ran before will load previous responses and any unfinished requests from the cache, allowing it to resume if all the required responses have not yet been retrieved.frompmawimportPushshiftAPIapi=PushshiftAPI()posts=api.search_submissions(subreddit="science",limit=700000,until=1613234822,safe_exit=True)print(f'{len(posts)}posts retrieved from Pushshift')Auntilvalue is required to load previous responses / requests when using non-id search, asuntilis set to the current time when thesearchmethod is called, which would result in a different set of parameters then when you last ran the search despite all other parameters being the same.Loading Cache with KeyTheResponseclass has a staticmethodload_cachethat allows you to pass akeyvalue andcache_dirto load responses from the cache into an instance ofResponse.frompmawimportResponsecache_key='a904e22ea572a8c0109b7bc5330528e4'cache_dir='./cache'resp=Response.load_cache(cache_key,cache_dir)BenchmarksBenchmark NotebookPMAW and PSAW ComparisonCompletion TimeA benchmark comparison was performed to determined the completion time for different size requests, ranging from 1 to 390,000 requested posts. This will allow us to determine which Pushshift wrappers and rate-limiting methods are best for different request sizes.Default parameters were used for eachPMAWrate-limit configuration as well as the defaultPSAWconfiguration, which does not provide multiple rate-limit implementations.For the first benchmark test we compare the completion times for all possiblePMAWrate-limiting configurations withPSAWfor up to 16,000 requested posts. We can see that the three most performant rate-limiting settings forPMAWare rate-averaging, and exponential backoff with full or equal jitter.We ran this second benchmark increasing up to 390,000 requested posts, excluding the least performantPMAWrate-limiting configurations. From this benchmark, we can see thatPMAWwas on average 1.79x faster thanPSAWat 390,625 posts retrieved. The total completion time for 390,625 posts withPSAWwas 2h38m, while the average completion time was 1h28m forPMAW.Number of RequestsWe also compare the number of required requests for each of the threePMAWrate-limit configurations. From this comparison, we can see that for 390,625 requested posts rate-averaging made 33.60% less API requests than exponential backoff.Memory Safety (Cache)A benchmark test was performed for the memory safety feature (mem_safe=True) to see the impact of caching responses has on the completion time, memory use, and max memory use while running requests for different limits.We can see that when memory safety was enabled, the completion time for 390,000 posts was 17.11% slower than when this feature was disabled and responses were not being cached, finishing in 1h30m instead of 1h17m.When memory safety is enabled responses start being cached after 20 checkpoints (defaultfile_checkpoint=20), equivalent to approximately 20,000 responses, causing the memory use to level out around 170MB of memory. Enabling memory safety allows us to use 90.97% less memory is used than when it is disabled, with the non-cached responses using 1.9GB of memory when 390,000 posts were retrieved. It's clear to see that we could easily trigger an out of memory error if we were to retrieve millions of submissions with memory safety disabled.We compare the maximum memory use during data retrieval as well. Once again, around the 20,000 response mark, the two methods diverge as responses begin to be added to the cache. For 390,000 posts, the maximum memory use when memory safety was enabled was 58.2% less than when it was disabled (797MB vs 1.9GB).Deprecated ExamplesThese examples are forpmawversion <=0.1.3.CommentsSearch Commentscomments=api.search_comments(subreddit="science",limit=1000)Search Comments by IDscomment_ids=['gjacwx5','gjad2l6','gjadatw','gjadc7w','gjadcwh','gjadgd7','gjadlbc','gjadnoc','gjadog1','gjadphb']comments_arr=api.search_comments(ids=comment_ids)Search Comment IDs by Submission IDpost_ids=['kxi2w8','kxi2g1','kxhzrl','kxhyh6','kxhwh0','kxhv53','kxhm7b','kxhm3s','kxhg37','kxhak9']comment_id_dict=api.search_submission_comment_ids(ids=post_ids)SubmissionsSearch Submissionssubmissions=api.search_submissions(subreddit="science",limit=1000)Search Submissions by IDspost_ids=['kxi2w8','kxi2g1','kxhzrl','kxhyh6','kxhwh0','kxhv53','kxhm7b','kxhm3s','kxhg37','kxhak9']posts_arr=api.search_submissions(ids=post_ids)LicensePMAWis released under the MIT License. See theLICENSEfile for more
details.