Package: wiki_tool_python
wiki_tool_pythonScript to perform various tasks with websites (wikiprojects) onMediaWikiengine using MediaWiki API.InstallationInstall Python 3.8 or higher, installpoetry, runpoetry install --no-dev.Then you can just runpoetry run COMMANDto run specific commands under python virtual environment created by poetry.Or you can enter poetry shell (by runningpoetry shell) and then type script commands.Installation exampleAssuming Python 3.8 or higher and poetry are installed.Initialize and update virtual environment (assuming you are in the folder with this README file):poetryinstall--no-devRun script:poetryrunpythonwiki_tool_python/wikitool.py--helpWindows installation exampleAssuming Python 3.8 or higher is installed.Install poetry (in Windows PowerShell):(Invoke-WebRequest-Urihttps://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py-UseBasicParsing).Content|pythonTo install or update libraries, run batch fileupdate.bat.To run script to download images, run batch filedownload_images.batand input URL, or run batch file with URL as first param, for example:download_images.bat https://absurdopedia.wiki/w.To run script to upload images, run batch fileupload_images.batand input your username, password and URL, or run batch file with URL as first param and type your username and password, for example:upload_images.bat https://absurdopedia.wiki/wand type username and password.To run script to get page list, run batch filelist_pages.batand input URL, or run batch file with URL as first param, for example:list_pages.bat https://absurdopedia.wiki/w.UsageGetting help--helpShow help message and exit. If this option is used with command, then help message for that specific command will be printed.Setting login and passwordSome commands require authentication. To tell login and password, use option--credentials LOGIN:PASSWORDor set environment variableMEDIAWIKI_CREDENTIALStoLOGIN:PASSWORD.Throttling requestsTo set interval between requests in seconds, use option--requests-interval SECONDS.Selecting MediaWiki versionTo select MediaWiki version, use option--mediawiki-version VERSION. Versions1.31and1.19are allowed, but1.19is not fully supported. Version1.31is default.Overwriting User-AgentTo setUser-Agentheader, use option--user-agent STRING. User-agent is set toWikiToolPythonby default.Force loginTo log in if it is not mandatory for API method, use option--login.MediaWiki API URLParameterAPI_URLshould be URL of MediaWiki API without/api.php, for examplehttps://wow.gamepedia.comorhttps://absurdopedia.wiki/w.Commandlist-imagespython wiki_tool_python/wikitool.py list-images [OPTIONS] API_URLList wikiproject images (titles and URLs).For each image four lines are printed:WordFILE2.Image name on wikiproject (including namespace, for exampleFile:Ogyglo.png).Image URL on wikiproject.Processed file name to save image.The same format is used bydownload-imagescommand.Note: deleted images may be listed in command output.Note: in earlier versions output format was different.Commandlist-images: options--output-file FILENAMEText file to write image list (standard output is used by default)--api-limit INTEGERMaximum number of entries per API request (default value is 500).--confine-encoding TEXTEncoding to confine file name to (drop characters outside that encoding)Commandlist-images: examplepythonwiki_tool_python/wikitool.pylist-imageshttps://wow.gamepedia.comOutput:...FILE2
Achievement guild forgreatjusticerank2.png
https://gamepedia.cursecdn.com/wowpedia/d/d6/Achievement_guild_forgreatjusticerank2.png
02986-Achievement guild forgreatjusticerank2.png...Commandlist-category-imagespython wiki_tool_python/wikitool.py list-category-images [OPTIONS] API_URL CATEGORYList wikiproject images (titles and URLs) from specific category in the same format aslist-images.Commandlist-category-images: options--output-file FILENAMEText file to write image list (standard output is used by default)--api-limit INTEGERMaximum number of entries per API request (default value is 500).--api-image-ids-limit INTEGERMaximum number of image IDs per API request (default value is 50).--confine-encoding TEXTEncoding to confine file name to (drop characters outside that encoding)Commandlist-category-images: examplepythonwiki_tool_python/wikitool.pylist-category-imageshttps://wow.gamepedia.com"Category:Images with watermarks"Output:...FILE2
Quartermaster Alcorn - Beta.jpg
https://gamepedia.cursecdn.com/wowpedia/2/2a/Quartermaster_Alcorn_-_Beta.jpg
00897-Quartermaster Alcorn - Beta.jpg...Commandlist-pagespython wiki_tool_python/wikitool.py list-pages [OPTIONS] API_URLList wikiproject page names of all wikiproject namespaces.Each page name is written on separate line.Note: deleted pages may be listed in command output (we have not checked for this yet).Commandlist-pages: options--output-file FILENAMEText file to write page names (standard output is used by default)--api-limit INTEGERMaximum number of entries per API request (default value is 500).Commandlist-pages: examplepythonwiki_tool_python/wikitool.pylist-pageshttps://wow.gamepedia.comOutput:...Talk:Alchemist (Warcraft III)
Talk:Alchemist Finklestein
Talk:Alchemist Pestlezugg
Talk:Alchemist Stone
Talk:Alchemy
Talk:Alchemy Lab
Talk:Alchemy proficiencies
Talk:Alchemy recipes
Talk:Alchemy recipes/Grand Master
Talk:Alchemy trainers)...Commandlist-namespace-pagespython wiki_tool_python/wikitool.py list-namespace-pages [OPTIONS] API_URL NAMESPACEList wikiproject page names of namespace (NAMESPACEshould be integer ID of namespace).Each page name is written on separate line.Note: deleted pages may be listed in command output (we have not checked for this yet).Commandlist-namespace-pages: options--output-file FILENAMEText file to write page names (standard output is used by default)--api-limit INTEGERMaximum number of entries per API request (default value is 500).Commandlist-namespace-pages: examplepythonwiki_tool_python/wikitool.pylist-namespace-pageshttps://wow.gamepedia.com1Output:...Talk:"Cookie" McWeaksauce
Talk:"Dirty" Larry
Talk:"Dirty" Michael Crowe
Talk:"Let the Fires Come!"/Notes
Talk:"Let the Fires Come!" (Alliance)
Talk:"Let the Fires Come!" (Horde)
Talk:"Red" Jack Findle
Talk:'Tis the Season
Talk:1.11.0 API changes (Iriel)
Talk:1.8.4
Talk:10-day Free Trial
Talk:19 Pound Catfish
Talk:1st Legion...Commandlist-deletedrevspython wiki_tool_python/wikitool.py list-deletedrevs [OPTIONS] OUTPUT_DIRECTORY API_URLList deleted revisions from wikiproject in JSON format, output may be splitted to many JSON files, files are saved toOUTPUT_DIRECTORY.Note: this command requires authentication.Note: files will be named:entry-N.json, whereNis number of file, starting from 0.Commandlist-deletedrevs: options--all-namespaces BOOLEANBoolean value,TRUEto list for all namespaces,FALSEfor main namespace only.--file-entry-num INTEGERNumber of entries per JSON file.--api-limit INTEGERMaximum number of entries per API request (default value is 500).Commanddelete-pagespython wiki_tool_python/wikitool.py delete-pages [OPTIONS] FILTER_EXPRESSION API_URLDelete pages matching regular expressionFILTER_EXPRESSION.Note: this command requires authentication.Note: this command deletes pages, use it with caution!Note: regular expressions are parsed with modulereof python standard library. Seere documentationfor details.Commanddelete-pages: options--reason TEXTDeletion reason.--exclude-expression TEXTAdditional expression to exclude pages matching it from deletion.--first-page TEXTFirst page to delete (start deletion from it).--first-page-namespace INTEGERNamespace of first page to delete.--api-limit INTEGERMaximum number of entries per API request (default value is 500).--namespace INTEGERNamespace to search pages for deletion (this option can be used multiple times to add multiple namespaces).Commandedit-pagespython wiki_tool_python/wikitool.py edit-pages [OPTIONS] FILTER_EXPRESSION NEW_TEXT API_URLEdit pages matching regular expressionFILTER_EXPRESSION, replacing their content with new textNEW_TEXT.Note: this command requires authentication.Note: this command edits pages, use it with caution!Note: regular expressions are parsed with modulereof python standard library. Seere documentationfor details.Commandedit-pages: options--reason TEXTEdit reason.--exclude-expression TEXTAdditional expression to exclude pages matching it from editing.--first-page TEXTFirst page to edit (start editing from it).--first-page-namespace INTEGERNamespace of first page to edit.--api-limit INTEGERMaximum number of entries per API request (default value is 500).--namespace INTEGERNamespace to search pages for deletion (this option can be used multiple times to add multiple namespaces).Commandedit-pages-clone-interwikispython wiki_tool_python/wikitool.py edit-pages-clone-interwikis [OPTIONS] API_URL OLD NEWAdd interwikiNEWto pages that contain interwikiOLD, but do not contain interwikiNEWyet.For example, ifNEWisen-gb, andOLDisen, than:[[en:Jaina Proudmoore]]will be replaced with:[[en:Jaina Proudmoore]]
[[en-gb:Jaina Proudmoore]]This command may be used, for example, if your wiki contains another language version, and this version has forked, and you need to add interwikis to fork to pages that already contain old version.Note: this command requires authentication.Note: this command edits pages, use it with caution!Commandedit-pages-clone-interwikis: options--reason TEXTEdit reason.--exclude-expression TEXTAdditional expression to exclude pages matching it from editing.--first-page TEXTFirst page to edit (start editing from it).--first-page-namespace INTEGERNamespace of first page to edit.--api-limit INTEGERMaximum number of entries per API request (default value is 500).Commandreplace-linkspython wiki_tool_python/wikitool.py replace-links [OPTIONS] API_URL OLD NEWReplace links to pageOLDby links to pageNEW.For example, ifNEWisWikiproject administrator, andOLDisAdministrator, than:Arthur Brahmaguptov was [[administrator]] of [[Encyclopedia Dramatica]].will be replaced with:Arthur Brahmaguptov was [[Wikiproject administrator|administrator]] of [[Encyclopedia Dramatica]].Note: this command requires authentication.Note: this command edits pages, use it with caution!Commandreplace-links: options--reason TEXTEdit reason.--api-limit INTEGERMaximum number of entries per API request (default value is 500).Command:replace-links: examplepython./wiki_tool_python/wikitool.pyreplace-linkshttp://wikireality.ru/wАдминистратор"Администратор википроекта"Commanddownload-imagespython wiki_tool_python/wikitool.py download-images [OPTIONS] LIST_FILE DOWNLOAD_DIRDownload images listed in fileLIST_FILE(file should be in the same format as output oflist-imagescommand) to directoryDOWNLOAD_DIR.Each image is downloaded from URL given in file and saved to file with name given in file (for example,File:Ogyglo.pngmay be saved as00500-Ogyglo.png).Note: progress bar is displayed.Note: if image can not be downloaded, warning is printed, but process is not stopped.Commanddownload-images: optionsNo specific options.Commanddownload-images: examplepythonwiki_tool_python/wikitool.pydownload-imagesimages.txtwiki_imagesWhereimages.txtis list of images.Output:Progress bar is displayed.Images are downloaded to directorywiki_images.Commandupload-imagepython wiki_tool_python/wikitool.py upload-image [OPTIONS] FILE_NAME FILE API_URLUpload imageFILEto wiki with new nameFILE_NAME.Note: this command requires authentication.Commandupload-image: optionsNo specific options.Commandupload-image: examplepythonwiki_tool_python/wikitool.pyupload-imageNVGogol.pngPic9231i9149i.pnghttps://absurdopedia.wiki/wCommandupload-imagespython wiki_tool_python/wikitool.py upload-images [OPTIONS] LIST_FILE DOWNLOAD_DIR API_URLUpload images listed in fileLIST_FILE(file should be in the same format as output oflist-imagescommand) from directoryDOWNLOAD_DIRto wiki.Note: this command requires authentication.Note: progress bar is displayed.Note: if image can not be upload, warning is printed, but process is not stopped.Commandupload-images: options--skip-nonexistent/--no-skip-nonexistentDo not fail on non-existent files, skip them insteadCommandupload-images: examplepythonwiki_tool_python/wikitool.pyupload-imagesimages.txtwiki_imageshttps://absurdopedia.wiki/wWhereimages.txtis list of images.Output:Progress bar is displayed.Commandvotecountpython wiki_tool_python/wikitool.py votecount [OPTIONS] API_URL USER_LIST_FILEGet user edit count for users from fileUSER_LIST_FILEand calculate vote power (according toWikireality rules).Commandvotecount: options--api-limit INTEGERMaximum number of entries per API request (default value is 500).--namespacefile FILENAMEJSON file to read namespaces data from.--start [%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S]Start date for counting edits.--end [%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S]End date for counting edits.--output-format [txt|mediawiki|json]Output data format (text, MediaWiki table or JSON).--redirect-regex-text TEXTRegular expression to detect redirect creation.Commandvotecount: exampleNamespace filenamespaces.jsonis:{"edit_weights":{"0":0.04,"1":0.003,"4":0.015,"5":0.003,"6":0.03,"7":0.003,"8":0.015,"9":0.003,"10":0.015,"11":0.003,"14":0.015,"15":0.003},"page_weights":{"0":0.6}}User list file../data/votecount/voters.txtis:Arbnos
Arsenal
Cat1987pythonwiki_tool_python/wikitool.py--mediawiki-version1.19votecount--start2018-07-12--end2018-10-12--namespacefile../data/votecount/namespaces.json--output-formattxthttp://wikireality.ru/w../data/votecount/voters.txtOutput:Processing user Arbnos...
Processing user Arsenal...
Processing user Cat1987...
User Arbnos
N0: 0
N1: 0
N4: 0
N5: 0
N6: 0
N7: 0
N8: 0
N9: 0
N10: 0
N11: 0
N14: 0
N15: 0
NewPages: 0
VotePower: 0.0

User Arsenal
N0: 20
N1: 0
N4: 9
N5: 0
N6: 0
N7: 0
N8: 0
N9: 0
N10: 2
N11: 0
N14: 0
N15: 0
NewPages: 2
VotePower: 2.165

User Cat1987
N0: 30
N1: 0
N4: 1
N5: 0
N6: 3
N7: 0
N8: 1
N9: 0
N10: 0
N11: 0
N14: 0
N15: 0
NewPages: 0
VotePower: 1.32...Commandlist-directory-pagespython wiki_tool_python/wikitool.py list-directory-pages [OPTIONS] INPUT_DIRECTORY OUTPUT_FILEWrite list of.txtfile paths in directory to JSON file.Directories are handled recursively.Commandlist-directory-pages: exampleThere are files../data/wiki-pages/Test.txtand../data/wiki-pages/Test/Test1.txt.python./wiki_tool_python/wikitool.pylist-directory-pages../data/wiki-pages/../data/wiki-pages-list.jsonFile../data/wiki-pages-list.jsonwill contain the following text:["Test.txt","Test/Test1.txt"]Commandlist-directory-pages: optionsNo specific options.Commandupload-pagespython wiki_tool_python/wikitool.py upload-pages [OPTIONS] API_URL INPUT_DIRECTORY LIST_FILECreate or overwrite pages from files in input directory listed in list file.Files with extension are uploaded as pages, page title for file is prefix concatenated with file path (as written in list file) without extension.Note: this command requires authentication.Note: list file format is either JSON list (same as of file generated bylist-directory-pagescommand) or JSON dictionary (keys are page pathes, values are page titles).Commandupload-pages: example 1There are files../data/wiki-pages/Test.txtand../data/wiki-pages/Test/Test1.txt.File../data/wiki-pages-list.jsoncontains the following text:["Test.txt","Test/Test1.txt"]Command is:python./wiki_tool_python/wikitool.pyupload-pages--prefix="User:ArtushakBot/"--summary="Test page upload"https://genepedia.ru../data/wiki-pages/../data/wiki-pages/wiki-pages-list.jsonThe following pages will be created:User:ArtushakBot/Testwith text from../data/wiki-pages/Test.txtUser:ArtushakBot/Test/Test1with text from../data/wiki-pages/Test/Test1.txtCommandupload-pages: example 2There are files../data/wiki-pages/1.txtand../data/wiki-pages/2.txt.File../data/wiki-pages-list.jsoncontains the following text:{"1.txt":"Page 1","2.txt":"Page 2"}Command is:python./wiki_tool_python/wikitool.pyupload-pages--dictionary--prefix="User:ArtushakBot/"--summary="Test page upload"https://genepedia.ru../data/wiki-pages/../data/wiki-pages/wiki-pages-list.jsonThe following pages will be created:User:ArtushakBot/Page 1with text from../data/wiki-pages/Page 1User:ArtushakBot/Page 2with text from../data/wiki-pages/Page 2Commandupload-pages: options--dictionary / --no-dictionaryUse dictionary file of pathes and titles instead of file list.--extended-dictionary / --no-extended-dictionaryRead dictionary values as dictionaries withtitleandpathkeys.--prefix TEXTPrefix for page titles.--summary TEXTEdit summary.--first-page INTEGERPage number to start with.--mode [append|overwrite]Whether to append text from file to old text on existing pages, or overwrite old text with text from file.Commandgenerate-import-scriptpython wiki_tool_python/wikitool.py list-directory-pages [OPTIONS] LIST_FILE INPUT_DIRECTORY OUTPUT_SCRIPT_FILE LOG_FILEWrite bash script to import pages from list file using MediaWiki maintenance scriptimportTextFiles.php(seeMediaWiki help). It can be uploaded to server along with page files and then run to import to them to website to avoid API overhead.LIST_FILEis path to page list fileon local machine.INPUT_DIRECTORYis parent path for page text fileson server, path should be absolute or relative to script working directory.LOG_FILEis path to output text log fileon server, path should be absolute or relative to script working directory.Note: server access (for example, SSH) is required for this method of page uploading.Commandgenerate-import-script: options--prefix TEXTPrefix for page titles.--rc / --no-rcAdd--rcoption to output script.--bot / --no-botAdd--botoption to output script.--user TEXTAdd -u option to output script.--summary TEXTAdd -s option to output script.--maintenance-directory TEXTDirectory withimportTextFiles.phpscripton server, path should be absolute or relative to script working directory.--show-progress-bar / --no-show-progress-barAdd progress bar display to output script--first-page INTEGERFirst page number to addCommandgenerate-import-script: exampleThere are files../data/wiki-pages/Test.txtand../data/wiki-pages/Test/Test1.txt.File../data/wiki-pages-list.jsoncontains the following text:["Test.txt","Test/Test1.txt"]Command is:python./wiki_tool_python/wikitool.pygenerate-import-script--prefix="User:ArtushakBot/"--summary="Test page upload"--bot--maintenance-directory="/srv/http/mediawiki/w/maintenance"../data/wiki-pages/wiki-pages-list.json./data/wiki-pages../data/importPages.sh./data/log.txtFile../data/importPages.shwill contain following text:#!/bin/bashphp/srv/http/mediawiki/w/maintenance/importTextFiles.php--rc--bot-s'Test page upload'--prefixUser:ArtushakBot/data/wiki-pages/Test.txt>>./data/log.txt2>&1echo-ne'[##########          ]         1 /         2\r'php/srv/http/mediawiki/w/maintenance/importTextFiles.php--rc--bot-s'Test page upload'--prefixUser:ArtushakBot/Test/data/wiki-pages/Test/Test1.txt>>./data/log.txt2>&1echo-ne'[####################]         2 /         2\r'Main examplepoetryinstall--no-dev
mkdir../absurdopedia
mkdir../absurdopedia/download
poetryrunpythonwiki_tool_python/wikitool.pylist-images--output-file../absurdopedia/image.txthttps://absurdopedia.wiki/w
poetryrunpythonwiki_tool_python/wikitool.pydownload-images../absurdopedia/image.txt../absurdopedia/downloadFilesmediawiki.pycontains exceptions and classes to interact with MediaWiki API. Abstract base class is namedMediaWikiAPI, implementations for specific MediaWiki versions are subclassed from it.wikitool.pycontains commands described in this file. To parse them,Clickis used.Special thanksMediaWikidevelopers.Pythondevelopers.Poetrydevelopers.requestsdevelopers.Clickdevelopers.Рыцарь (Knight) aka Riddari aka Тэйтанка-птекила (Teitanka-ptekila).Wikirealityusers, especially members of Dimetr's Telegram chat.Other folks from Absurdopedia, russian version of Uncyclopedia (absurdopedia.wikiandabsurdopedia.net)Other cool folks.