Package: archean
Table of ContentsSupported SystemsAboutInstallationUsage NotesFAQs1. Why is only MongoDB supported as a Database?2. The parser only extracts information from the latest version of the pages. Why?3. The parser is only extracts Film Infobox. Why is that so? Can you extend the support to other parts of Wikipedia articles?4. Is there any plan to extend this parser to other Infobox types as well?Supported SystemsWindows 10Linux (tested on ArchLinux, Manjaro, Ubuntu)AboutArchean is a tool to process Wikipedia dumps and extract required information from them. The tool contains several files:wiki_downloader.pywiki_parser.pydb_writer.pycleanup.pywiki_downloaderis used to download the dumps from a wikipedia dump directory.wiki_parseris the brain of the project. The module houses the logic for processing the Wikipedia dumps and extract information.cleanup.pyis the file for cleaning the extracted content to be in structured form. Lastly,db_writeris an additional tool in case the JSON file created from the dumps need to be written into the database. The supported database is MongoDB.Installationpip install archeanUsage NotesThe parser accepts a few parameters that can be specified during script invokation. These are lised below:archean: Check if bz2 files are present in the same directory. If yes, it should start unpacking and processing in folder named 'extracts' in present working directory. If not, then ask user to specify remote-directory to download. No DB operations to be done since DB details are not provided.archean  --version: Should display the version of the applicationarchean –-remote-dir='REMOTE': Should download the bz2 files, from REMOTE, in the present working directory/the folder user executed the command in. processing of files to be done as well and json files to be placed inside a sub-folder named 'extracts'archean –-extraction-dir='EXTRACTION_DIR': Check if bz2 files are present in the same directory. If yes, processing of files to be done as well and JSON files are to be placed after creating a subfolder EXTRACTION_DIR in the present working directory. If not, then ask user to specify remote-directory. No DB operations to be done since DB details are not provided.archean –-remote-dir='REMOTE' –-extraction-dir='EXTRACTION_DIR': Should download the bz2 files, from REMOTE, in the present working directory/the folder user executed the command in. processing of files to be done as well and JSON files are to be placed after creating a subfolder EXTRACTION_DIR in the present working directory. No DB operations to be done since DB details are not provided.archean –-remote-dir='' –-download-only: Should download the bz2 files in the present working directory/the folder user executed the command in. No processing of the files and no DB operation to be performed since download-only argument is provided.archean –-download-only: Throw an error since no remote directory (remote-dir) is provided.archean --host='HOST' --port='PORT' --db='DB' --collection='COLLECTION' --user='USER' --password='PASSWORD': Assume processing of bz2 files is to be done and files are in the present working directory. check if bz2 files are present in the same directory. If yes, it should start unpacking and processing in 'extracts' folder. After JSON files are obtained, connect with the database 'DB' located at HOST using USER and PASSWORD, and insert JSON data into the COLLECTION.archean --host='HOST' --port='PORT' --db='DB' --collection='COLLECTION' --user='USER' --password='PASSWORD' --extraction-dir='EXTRACTION_DIR': Does all operations as above, but JSON files are created in the folder specified in 'extraction-dir' argument.FAQs1. Why is only MongoDB supported as a Database?Wikipedia is not a structured information collection. The information extracted from the Wikipedia might be missing some information in case of one article but might be present in another article. In such cases, NoSQL databases become an obvious choice of data storage. Hence MongoDB was chosen.2. The parser only extracts information from the latest version of the pages. Why?Wikipedia has a lot of information. It keeps the edit history of the pages in its archive but since any project is less likely to involve processing of old version data, thedownloaderhas been kept at minimum to download only the latest version.3. The parser is only extracts Film Infobox. Why is that so? Can you extend the support to other parts of Wikipedia articles?Infoboxes are great summary sections in Wikipedia pages. They can provide answers to most common queries in a giffy. Hencewiki-parseris created to parse Infoboxes first.
Why the choice of Film infobox was made is simply because it is easy to judge the validity of  parsed information during development phase. Also because we all love movies ;)4. Is there any plan to extend this parser to other Infobox types as well?Definitely! There is so much to be done in the project. Infobox for books information, for countries, for music, for magazines,and so much more are there to cater the project to.