Package: bdfrx
Bulk Downloader for Reddit xThis is a tool to download submissions from Reddit. BDFRx is flexible and can be used in scripts if needed through an extensive command-line interface.List of currently supported sourcesIf you wish to open an issue, please readthe guide on opening issuesto ensure that your issue is clear and contains everything it needs to for the developers to investigate.Included in this README are a few example Bash tricks to get certain behaviour. For that, seeCommon Command Tricks.InstallationBulk Downloader for Reddit xneeds Python version 3.9 or above. Please update Python before installation to meet the requirement.Approximately Dec 10, 2023 the minimum version is expected to move to 3.10 at the six month mark of Debian 12's release.Then, you can install it viapipxwith:pipxinstallbdfrxor via pip with:python3-mpipinstallbdfrx--upgradeTo update BDFRx, run the above command again for pip orpipx upgrade bdfrxfor pipx installations.To check your version of BDFRx, runbdfrx --versionTo install shell completions, runbdfrx completionSource codeIf you want to use the source code or make contributions, refer toCONTRIBUTINGDifferences fromBDFRBDFRx differs from the base BDFR in a few ways:Does not contain the Archive and Clone modes (If you require these modes you can try v0.9.1, providedas isor use the baseBDFRproject)Option to use an Sqlite3 database to store hashes, links and post ID's to be filtered in future runsUses thebdfrxconfig directory by defaultReddit API ChangesStarting July 1st 2023 (along with the other changes) rate limits will beappliedper client idrather than by client id/user combination. Therefore it is no longer tenable to ship BDFRx with an installed client id as it will cause rate limiting for all users.The shipped client id will no longer be valid after the changes.To create your own for use with BDFRx gohereand click "create an app". It's recommended to choose "Installed App" and use "http://localhost:7634" as the redirect url.Once you have your personal client id add it to yourconfig.cfgat the location in theconfigurationsection.UsageBDFRx works by taking submissions from a variety of "sources" from Reddit and then parsing them to download. These sources might be a subreddit, multireddit, a user list, or individual links. These sources are combined and downloaded to disk, according to a naming and organisational scheme defined by the user.The sole mode of BDFRx is download. Thedownloadcommand will download the resource linked in the Reddit submission, such as the images, video, etc.After installation, run the program from any directory as shown below:bdfrxdownloadHowever, this command is not enough. You should chain parameters inOptionsaccording to your use case. Don't forget that some parameters can be provided multiple times. Some quick reference commands are:bdfrxdownload./path/to/output--subredditPython-L10bdfrxdownload./path/to/output--userreddituser--submitted-L100bdfrxdownload./path/to/output--userme--saved--authenticate-L25--file-scheme"{POSTID}"bdfrxdownload./path/to/output--subreddit"Python, all, mindustry"-L10--make-hard-linksAlternatively, you can pass options through a YAML file.bdfrxdownload./path/to/output--optsmy_opts.yamlFor example, running it with the following fileskip:[mp4,avi]file_scheme:"{UPVOTES}_{REDDITOR}_{POSTID}_{DATE}"limit:10sort:topsubreddit:-EarthPorn-CityPornwould be equilavent to (take note that in YAML there isfile_schemeinstead offile-scheme):bdfrxdownload./path/to/output--skipmp4--skipavi--file-scheme"{UPVOTES}_{REDDITOR}_{POSTID}_{DATE}"-L10-Stop--subredditEarthPorn--subredditCityPornAny option that can be specified multiple times should be formatted like subreddit is above.In cases when the same option is specified both in the YAML file and in as a command line argument, the command line argument takes priorityOptionsdirectoryThis is the directory to which BDFRx will download and place all files--authenticateThis flag will make BDFRx attempt to use an authenticated Reddit sessionSeeAuthenticationfor more details--configIf the path to a configuration file is supplied with this option, BDFRx will use the specified configSeeConfiguration Filesfor more details--dbConnects to a sqlite3 db to save hashes, paths and links to be reusedAutomatically sets no-dupes to enabled using saved hashesAllows hard links to be created with paths based on saved hashesSaves links so they are skipped from downloading in the futureComplements exclude-id options but does not require them--db-fileAllows to specify a location of the sqlite3 db rather than storing in the config directory--disable-moduleCan be specified multiple timesDisables certain modules from being usedSeeDisabling Modulesfor more information and a list of module names--downvotedThis will use a user's downvoted posts as a source of posts to scrapeThis requires an authenticated Reddit instance, using the--authenticateflag, as well as--userset tome--exclude-idThis will skip the download of any submission with the ID providedCan be specified multiple times--exclude-id-fileThis will skip the download of any submission with any of the IDs in the files providedCan be specified multiple timesFormat is one ID per line--file-schemeSets the scheme for filesDefault is{REDDITOR}_{TITLE}_{POSTID}SeeFolder and File Name Schemesfor more details--filename-restriction-schemeCan be:windows,linuxTurns off the OS detection and specifies which system to use when making filenamesSeeFilesystem Restrictions--folder-schemeSets the scheme for foldersDefault is{SUBREDDIT}SeeFolder and File Name Schemesfor more details--ignore-userThis will add a user to ignoreCan be specified multiple times--include-id-fileThis will add any submission with the IDs in the files providedCan be specified multiple timesFormat is one ID per line-L, --limitThis is the limit on the number of submissions retrieveDefault is max possibleNote that this limit applies toeach source individuallye.g. if a--limitof 10 and three subreddits are provided, then 30 total submissions will be scrapedIf it is not supplied, then BDFRx will default to the maximum allowed by Reddit, roughly 1000 posts.This cannot be bypassed.-l, --linkThis is a direct link to a submission to download, either as a URL or an IDCan be specified multiple times--logThis allows one to specify the location of the logfileThis must be done when running multiple instances of BDFRx, seeMultiple Instancesbelow--make-hard-linksThis flag will create hard links to an existing file when a duplicate is downloaded in the current runThis will make the file appear in multiple directories while only taking the space of a single instance--max-wait-timeThis option specifies the maximum wait time for downloading a resourceThe default is 120 secondsSeeRate Limitingfor details--min-scoreThis skips all submissions which have fewer than specified upvotes--max-scoreThis skips all submissions which have more than specified upvotes--min-score-ratioThis skips all submissions which have lower than specified upvote ratio--max-score-ratioThis skips all submissions which have higher than specified upvote ratio-m, --multiredditThis is the name of a multireddit to add as a sourceCan be specified multiple timesThis can be done by using-mmultiple timesMultireddits can also be used to provide CSV multireddits e.g.-m "chess, favourites"The specified multireddits must all belong to the user specified with the--useroption--no-dupesThis flag will skip writing a file to disk if that file was already downloaded in the current runThis is calculated by MD5 hash--optsLoad options from a YAML file.Has higher prority than the global config file but lower than command-line arguments.Seeopts_example.yamlfor an example file.--savedThis option will make BDFRx use the supplied user's saved posts list as a download sourceThis requires an authenticated Reddit instance, using the--authenticateflag, as well as--userset tome--searchThis will apply the input search term to specific lists when scraping submissionsA search term can only be applied when using the--subredditand--multiredditflags--search-existingThis will make BDFRx compile the hashes for every file indirectoryThe hashes are used to skip duplicate files if--no-dupesis supplied or make hard links if--make-hard-linksis suppliedThe use of this option is highly discouraged due to inefficiency--skipThis adds file types to the download filter i.e. submissions with one of the supplied file extensions will not be downloadedCan be specified multiple times--skip-domainThis adds domains to the download filter i.e. submissions coming from these domains will not be downloadedCan be specified multiple timesDomains must be supplied in the formexample.comorimg.example.com--skip-subredditThis skips all submissions from the specified subredditCan be specified multiple timesAlso accepts CSV subreddit names-S, --sortThis is the sort type for each applicable submission source supplied to BDFRxThis option does not apply to upvoted, downvoted or saved posts when scraping from these sourcesThe following options are available:controversialhot(default)newrelevance(only available when using--search)risingtop--submittedThis will use a user's submissions as a sourceA user must be specified with--user-s, --subredditThis adds a subreddit as a sourceCan be used mutliple timesThis can be done by using-smultiple timesSubreddits can also be used to provide CSV subreddits e.g.-m "all, python, mindustry"-t, --timeThis is the time filter that will be applied to all applicable sourcesThis option does not apply to upvoted, downvoted or saved posts when scraping from these sourcesThis option only applies if sorting by top or controversial.  See --sort for more detail.The following options are available:all(default)hourdayweekmonthyear--time-formatThis specifies the format of the datetime string that replaces{DATE}in file and folder naming schemesSeeTime Formatting Customisationfor more details, and the formatting scheme--upvotedThis will use a user's upvoted posts as a source of posts to scrapeThis requires an authenticated Reddit instance, using the--authenticateflag, as well as--userset tome-u, --userThis specifies the user to scrape in concert with other optionsWhen using--authenticate,--user mecan be used to refer to the authenticated userCan be specified multiple times for multiple usersIf downloading a multireddit, only one user can be specified-v, --verboseIncreases the verbosity of the programCan be specified multiple timesCommon Command TricksA common use case is for subreddits/users to be loaded from a file. BDFRx supports this via YAML file options (--opts my_opts.yaml).Alternatively, you can use the command-linexargsfunction.
For a list of usersusers.txt(one user per line), type:catusers.txt|xargs-L1echo--user|xargs-L50bdfrxdownload<ARGS>The part-L 50is to make sure that the character limit for a single line isn't exceeded, but may not be necessary. This can also be used to load subreddits from a file, simply exchange--userwith--subredditand so on.Authentication and SecurityBDFRx uses OAuth2 authentication to connect to Reddit if authentication is required. This means that it is a secure, token-based system for making requests. This also means that BDFRx only has access to specific parts of the account authenticated, by default only saved posts, upvoted posts, downvoted posts, and the identity of the authenticated account. Note that authentication is not required unless accessing private things like upvoted posts, downvoted posts, saved posts, and private multireddits.To authenticate, BDFRx will first look for a token in the configuration file that signals that there's been a previous authentication. If this is not there, then BDFRx will attempt to register itself with your account. This is normal, and if you run the program, it will pause and show a Reddit URL. Click on this URL and it will take you to Reddit, where the permissions being requested will be shown. Read this andconfirm that there are no more permissions than needed to run the program. You should not grant unneeded permissions; by default, BDFRx only requests permission to read your saved, upvoted, or downvoted submissions and identify as you.If the permissions look safe, confirm it, and BDFRx will save a token that will allow it to authenticate with Reddit from then on.Changing PermissionsMost users will not need to do anything extra to use any of the current features. However, if additional features such as scraping messages, PMs, etc are added in the future, these will require additional scopes. Additionally, advanced users may wish to use BDFRx with their own API key and secret. There is normally no need to do this, but itisallowed by BDFRx.The configuration file for BDFRx contains the API secret and key, as well as the scopes that BDFRx will request when registering itself to a Reddit account via OAuth2. These can all be changed if the user wishes, however do not do so if you don't know what you are doing. The defaults are specifically chosen to have a very low security risk if your token were to be compromised, however unlikely that actually is. Never grant more permissions than you absolutely need.For more details on the configuration file and the values therein, seeConfiguration Files.Folder and File Name SchemesThe naming and folder schemes for BDFRx are both completely customisable. A number of different fields can be given which will be replaced with properties from a submission when downloading it. The scheme format takes the form of{KEY}, whereKEYis a string from the below list.DATEFLAIRPOSTIDREDDITORSUBREDDITTITLEUPVOTESEach of these can be enclosed in curly bracket,{}, and included in the name. For example, to just title every downloaded post with the unique submission ID, you can use{POSTID}. Static strings can also be included, such asdownload_{POSTID}which will not change from submission to submission. For example, the previous string will result in the following submission file names:download_aaaaaa.pngdownload_bbbbbb.pngAt least one keymustbe included in the file scheme, otherwise an error will be thrown. The folder scheme however, can be null or a simple static string. In the former case, all files will be placed in the folder specified with thedirectoryargument. If the folder scheme is a static string, then all submissions will be placed in a folder of that name. In both cases, there will be no separation between all submissions.It is highly recommended that the file name scheme contain the parameter{POSTID}as this isthe only parameter guaranteed to be unique. No combination of other keys will necessarily be unique and may result in posts being skipped as BDFRx will see files by the same name and skip the download, assuming that they are already downloaded.ConfigurationThe configuration files are, by default, stored in the configuration directory for the user. This differs depending on the OS that BDFRx is being run on. For Windows, this will be:C:\Users\<User>\AppData\Local\BDFRx\bdfrxIf Python has been installed through the Windows Store, the folder will appear in a different place. Note that the hash included in the file path may change from installation to installation.C:\Users\<User>\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\Local\BDFRx\bdfrxOn Mac OSX, this will be:~/Library/Application Support/bdfrx.Lastly, on a Linux system, this will be:~/.config/bdfrx/The logging output for each run of BDFRx will be saved to this directory in the filelog_output.txt. If you need to submit a bug, it is this file that you will need to submit with the report.Configuration FileTheconfig.cfgis the file that supplies BDFRx with the configuration to use. At the moment, the following keysmustbe included in the configuration file supplied.client_idscopesThe following key may be required depending on the type of client id you created.client_secretThe following keys are optional, and defaults will be used if they cannot be found.backup_log_countmax_wait_timetime_formatdisabled_modulesfilename-restriction-schemeAll of these should not be modified unless you know what you're doing, as the default values will enable BDFRx to function just fine. A configuration is included in BDFRx when it is installed, and this will be placed in the configuration directory as the default.Most of these values have to do with OAuth2 configuration and authorisation. The keybackup_log_counthowever has to do with the log rollover. The logs in the configuration directory can be verbose and for long runs of BDFRx, can grow quite large. To combat this, BDFRx will overwrite previous logs. This value determines how many previous run logs will be kept. The default is 3, which means that BDFRx will keep at most three past logs plus the current one. Any runs past this will overwrite the oldest log file, called "rolling over". If you want more records of past runs, increase this number.Time Formatting CustomisationThe optiontime_formatwill specify the format of the timestamp that replaces{DATE}in filename and folder name schemes. By default, this is theISO 8601format which is highly recommended due to its standardised nature. If you don'tneedto change it, it is recommended that you do not. However, you can specify it to anything required with this option. The--time-formatoption supersedes any specification in the configuration fileThe format can be specified through theformat codesthat are standard in the Pythondatetimelibrary.Disabling ModulesThe individual modules of BDFRx, used to download submissions from websites, can be disabled. This is helpful especially in the case of the fallback downloaders, since the--skip-domainoption cannot be effectively used in these cases. For example, the Youtube-DL downloader can retrieve data from hundreds of websites and domains; thus the only way to fully disable it is via the--disable-moduleoption.Modules can be disabled through the command line interface for BDFRx or more permanently in the configuration file via thedisabled_modulesoption. The list of downloaders that can be disabled are the following. Note that they are case-insensitive.DirectDelayForRedditEromeGallery(Reddit Image Galleries)GfycatImgurPornHubRedgifsSelfPost(Reddit Text Post)VidbleVReddit(Reddit Video Post)YoutubeYtdlpFallback(Youtube DL Fallback)Rate LimitingThe optionmax_wait_timehas to do with retrying downloads. There are certain HTTP errors that mean that no amount of requests will return the wanted data, but some errors are from rate-limiting. This is when a single client is making so many requests that the remote website cuts the client off to preserve the function of the site. This is a common situation when downloading many resources from the same site. It is polite and best practice to obey the website's wishes in these cases.To this end, BDFRx will sleep for a time before retrying the download, giving the remote server time to "rest". This is done in 60 second increments. For example, if a rate-limiting-related error is given, BDFRx will sleep for 60 seconds before retrying. Then, if the same type of error occurs, it will sleep for another 120 seconds, then 180 seconds, and so on.The option--max-wait-timeand the configuration optionmax_wait_timeboth specify the maximum time BDFRx will wait. If both are present, the command-line option takes precedence. For instance, the default is 120, so BDFRx will wait for 60 seconds, then 120 seconds, and then move one.Note that this results in a total time of 180 seconds trying the same download. If you wish to try to bypass the rate-limiting system on the remote site, increasing the maximum wait time may help. However, note that the actual wait times increase exponentially if the resource is not downloaded i.e. specifying a max value of 300 (5 minutes), can make BDFRx pause for 15 minutes on one submission, not 5, in the worst case.Multiple InstancesBDFRx can be run in multiple instances with multiple configurations, either concurrently or consecutively. The use of scripting files facilitates this the easiest, either Powershell on Windows operating systems or Bash elsewhere. This allows multiple scenarios to be run with data being scraped from different sources, as any two sets of scenarios might be mutually exclusive i.e. it is not possible to download any combination of data from a single run of BDFRx. To download from multiple users for example, multiple runs of BDFRx are required.Running these scenarios consecutively is done easily, like any single run. Configuration files that differ may be specified with the--configoption to switch between tokens, for example. Otherwise, almost all configuration for data sources can be specified per-run through the command line.Running scenarios concurrently (at the same time) however, is more complicated. BDFRx will look to a single, static place to put the detailed log files, in a directory with the configuration file specified above. If there are multiple instances, or processes, of BDFRx running at the same time, they will all be trying to write to a single file. On Linux and other UNIX based operating systems, this will succeed, though there is a substantial risk that the logfile will be useless due to garbled and jumbled data. On Windows however, attempting this will raise an error that crashes the program as Windows forbids multiple processes from accessing the same file.The way to fix this is to use the--logoption to manually specify where the logfile is to be stored. If the given location is unique to each instance of BDFRx, then it will run fine.Filesystem RestrictionsDifferent filesystems have different restrictions for what files and directories can be named. Thesse are separated into two broad categories: Linux-based filesystems, which have very few restrictions; and Windows-based filesystems, which are much more restrictive in terms if forbidden characters and length of paths.During the normal course of operation, BDFRx detects what filesystem it is running on and formats any filenames and directories to conform to the rules that are expected of it. However, there are cases where this will fail. When running on a Linux-based machine, or another system where the home filesystem is permissive, and accessing a share or drive with a less permissive system, BDFRx will assume that thehomefilesystem's rules apply. For example, when downloading to a SAMBA share from Ubuntu, there will be errors as SAMBA is more restrictive than Ubuntu.The best option would be to always download to a filesystem that is as permission as possible, such as an NFS share or ext4 drive. However, when this is not possible, BDFRx allows for the restriction scheme to be manually specified at either the command-line or in the configuration file. At the command-line, this is done with--filename-restriction-scheme windows, or else an option by the same name in the configuration file.Manipulating LogfilesThe logfiles that BDFRx outputs are consistent and quite detailed and in a format that is amenable to regex. To this end, a number of bash scripts have beenincluded here. They show examples for how to extract successfully downloaded IDs, failed IDs, and more besides.Unsaving postsBack in BDFR v1 there was an option to unsave posts from your account when downloading, but it was removed from the core BDFR on v2 as it is considered a read-only tool. However, for those missing this functionality, a script was created that uses the log files to achieve this. There is info on how to use this on the README.md file on the scripts subdirectory.List of currently supported sourcesDirect links (links leading to a file)Delay for RedditEromeGfycatGif Delivery NetworkImgurReddit GalleriesReddit Text PostsReddit VideosRedgifsVidbleYouTubeAny source supported byYT-DLPshould be compatableContributingIf you wish to contribute, seeContributingfor more information.When reporting any issues or interacting with the developers, please follow theCode of Conduct.