Package: scribe-data
Wikidata and Wikipedia language data extractionScribe-Datacontains the scripts for extracting and formatting data fromWikidataandWikipediafor Scribe applications. Updates to the language keyboard and interface data can be done usingscribe_data/load/update_data.pyand the notebooks within thescribe_data/loaddirectory.[!NOTE]Thecontributingsection has information for those interested, with the articles and presentations infeatured byalso being good resources for learning more about Scribe.Scribe applications are available oniOS,Android(WIP) andDesktop(planned).ContentsProcessContributingEnvironment SetupSupported LanguagesFeatured ByProcess⇧scribe_data/load/update_data.pyand the notebooks within thescribe_data/loaddirectory are used to update all data forScribe-iOS, with this functionality later being expanded to updateScribe-AndroidandScribe-Desktopwhen they're active.The main data update process inupdate_data.pytriggersSPARQL queriesto query language data fromWikidatausingSPARQLWrapperas a URI. The autosuggestion process derives popular words fromWikipediaas well as those words that normally follow them for an effective baseline feature until natural language processing techniques are employed. Functions to generate autosuggestions are ran ingen_autosuggestions.ipynb. Emojis are further sourced fromUnicode CLDR, with this process being ran ingen_emoji_lexicon.ipynb.The ultimate goal is that this repository will house language packs that are periodically updated with newWikidatalexicographical data and data from other sources. These packs would then be available to download by users of Scribe applications.Contributing⇧Scribe usesMatrixfor communications. You're more than welcome tojoin us in our public chat roomsto share ideas, ask questions or just say hi :)Please see thecontribution guidelinesandWikidata and Scribe Guideif you are interested in contributing to Scribe-Data. Work that is in progress or could be implemented is tracked in theissuesandprojects.[!NOTE]Just because an issue is assigned on GitHub doesn't mean that the team isn't interested in your contribution! Feel free to writein the issuesand we can potentially reassign it to you.Those interested can further check the-next release-and-priority-labels in theissuesfor those that are most important, as well as those markedgood first issuethat are tailored for first time contributors.After your first few pull requests organization members would be happy to discuss granting you further rights as a contributor, with a maintainer role then being possible after continued interest in the project. Scribe seeks to be an inclusive and supportive organization. We'd love to have you on the team!Ways to Help⇧Reporting bugsas they're found 🐞Working onnew features✨Documentationfor onboarding and project cohesion 📝Adding language data toScribe-DataviaWikidata! 🗃️Road Map⇧The Scribe road map can be followed in the organization'sproject boardwhere we list the most important issues along with their priority, status and an indication of which sub projects they're included in (if applicable).Data Edits⇧[!NOTE]Please see theWikidata and Scribe Guidefor an overview ofWikidataand how Scribe uses it.Scribe does not accept direct edits to the grammar JSON files as they are sourced fromWikidata. Edits can be discussed and the queries themselves will be changed and ran before an update. If there is a problem with one of the files, then the fix should be made onWikidataand not on Scribe. Feel free to let us know that edits have been made byopening a data issueand we'll be happy to integrate them!Environment Setup⇧The development environment for Scribe-Data can be installed via the following steps:ForktheScribe-Data repo, clone your fork, and configure the remotes:[!NOTE]Consider using SSHAlternatively to using HTTPS as in the instructions below, consider SSH to interact with GitHub from the terminal. SSH allows you to connect without a user-pass authentication flow.To run git commands with SSH, remember then to substitute the HTTPS URL,https://github.com/..., with the SSH one,git@github.com:....e.g. Cloning now becomesgit clone git@github.com:<your-username>/Scribe-Data.gitGitHub also has their documentation on how toGenerate a new SSH key🔑# Clone your fork of the repo into the current directory.gitclonehttps://github.com/<your-username>/Scribe-Data.git# Navigate to the newly cloned directory.cdScribe-Data# Assign the original repo to a remote called "upstream".gitremoteaddupstreamhttps://github.com/scribe-org/Scibe-Data.gitNow, if you rungit remote -vyou should see two remote repositories named:origin(forked repository)upstream(Scribe-Data repository)UsePython venvto create the local development environment within your Scribe-Data directory:python3-mvenvvenv# make an environment venvpipinstall--upgradepip# make sure that pip is at the latest versionpipinstall-rrequirements.txt# install dependencies[!NOTE]
Feel free to contact the team in theData room on Matrixif you're having problems getting your environment setup!Supported Languages⇧Scribe's goal is functional, feature-rich keyboards and interfaces for all languages. Check theextract_transformdirectory for queries for currently supported languages and those that have substantial data onWikidata.The following table shows the supported languages and the amount of data available for each onWikidataand viaUnicode CLDRfor emojis:LanguagesNounsVerbsTranslations*Prepositions†Emoji KeywordsFrench17,0726,57267,652-2,488German102,8333,59367,6522102,898Italian8,6717367,652-2,457Portuguese5,43753667,652-2,327Russian194,4481267,652153,827Spanish39,1054,93067,652-3,134Swedish45,2594,50167,652-2,913*Given the currentbetastatus where words are machine translated.†Only for languages for which preposition annotation is needed.Featured By⇧Articles and Presentations on Scribe2024February:Presentation slidesfor Scribe's participation at theWikimedia Tech Safari Program2023August:Scribe-iOS final submission report for Google Summer of Code 2023June:Scribe-iOS development blog post on Nested UITableViews & Apple's built-in ViewControllers in app menuforGSoC '23March:Presentation slidesfor a talk atBerlin Hack and Tell(Hack of the month winner 🏆)2022August:Presentation slidesfor a session at the2022 Wikimania HackathonJuly:Presentation slidesfor a talk atCocoaHeads BerlinJuly:Video on ScribeforWikimedia Celtic Knot 2022June:Presentation slidesfor a talk with theLD4 Wikidata Affinity GroupJune:Scribefeatured for new developers onMediaWikiMay:Presentation slidesforWikimedia Hackathon 2022March:Blog postonScribe-iOSforWikimedia Tech News(DE/Tweet)March:Presentation slidesforWikidata Data Reuse Days 2022Powered ByContributorsMany thanks to all theScribe-Data contributors! 🚀Blog postsList of referenced postsBuilding a Recommendation System Using Neural Network EmbeddingsbyWillKoehrsenWikipedia Data Science: Working with the World’s Largest EncyclopediabyWillKoehrsenWikimedia Communities