Package: kwnlp-sql-parser
Kensho Wikimedia for Natural Language Processing - SQL Dump Parserkwnlp_sql_parser is a Python package for parsingWikipedia SQL dumpsinto CSVs.Quick Install (Requires Python >= 3.6)pipinstallkwnlp-sql-parserExamplesBasic UsageTo convert a Wikipedia MySQL/MariaDB dump into a CSV file, use theto_csvmethod of theWikipediaSqlDumpclass.  By default, the CSV file is created in the current directory and includes all of the columns and rows in the SQL dump file.importpandasaspdfromkwnlp_sql_parserimportWikipediaSqlDumpfile_path="/path/to/data/enwiki-20200920-page.sql.gz"wsd=WikipediaSqlDump(file_path)wsd.to_csv()df=pd.read_csv("enwiki-20200920-page.csv",keep_default_na=False,na_values=[""])print(df.head())page_idpage_namespacepage_titlepage_restrictionspage_is_redirectpage_is_newpage_randompage_touchedpage_links_updatedpage_latestpage_lenpage_content_modelpage_lang0100AccessibleComputingNaN100.331671202009030748512.020090e+1385485158694wikitextNaN1120AnarchismNaN000.786172202009200236132.020092e+1397926749488697wikitextNaN2130AfghanistanHistoryNaN100.062150202009091841382.020091e+1378386514990wikitextNaN3140AfghanistanGeographyNaN100.952234202009151009452.020091e+1378386516092wikitextNaN4150AfghanistanPeopleNaN100.574721202009170806442.020091e+1378386529395wikitextNaNSee the "Common Issues" section below for an explanation of the pandas read_csv kwargs.Filtering Rows and ColumnsIn some situations, it is convenient to filter the Wikipedia SQL dumps before writing to CSV.  For example, one might only be interested in the columnspage_idandpage_titlefor Wikipedia pages that are in the (Main/Article)namespace.importpandasaspdfromkwnlp_sql_parserimportWikipediaSqlDumpfile_path="/path/to/data/enwiki-20200920-page.sql.gz"wsd=WikipediaSqlDump(file_path,keep_column_names=["page_id","page_title"],allowlists={"page_namespace":["0"]})wsd.to_csv()df=pd.read_csv("enwiki-20200920-page.csv",keep_default_na=False,na_values=[""])print(df.head())page_idpage_title010AccessibleComputing112Anarchism213AfghanistanHistory314AfghanistanGeography415AfghanistanPeopleNote that you can also specifyblocklistsinstead ofallowlistsif it is more convenient for your use case.Common IssuesNot using string values in filtersAll values in the allowlists and blocklists should be strings.Pages with names treated as NullBe carefull when reading the CSVs in your chosen software. Some packages will treat the following page titles as null values instead of strings,https://en.wikipedia.org/wiki/NaNhttps://en.wikipedia.org/wiki/Nullhttps://en.wikipedia.org/wiki/NaIn pandas this can be handled by reading in the CSV using,df=pd.read_csv("enwiki-20200920-page.csv",keep_default_na=False,na_values=[""])Supported Tableshttps://www.mediawiki.org/wiki/Manual:Category_tablehttps://www.mediawiki.org/wiki/Manual:Categorylinks_tablehttps://www.mediawiki.org/wiki/Manual:Page_tablehttps://www.mediawiki.org/wiki/Manual:Page_props_tablehttps://www.mediawiki.org/wiki/Manual:Pagelinks_tablehttps://www.mediawiki.org/wiki/Manual:Redirect_tableLicenseLicensed under the Apache 2.0 License. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.Copyright 2020-present Kensho Technologies, LLC. The present date is determined by the timestamp of the most recent commit in the repository.