Package: itemsubjector
ItemSubjectorThe purpose of this command-line tool is to add main subject statements to Wikidata
items based on a heuristic matching the subject with the title of the item.The tool running in PAWS adding manually found main subject QIDsItemsubjector running GNU Screen on a Toolforge bastion with --limit 100000 and
--sparql matching the WHO list of essential medicines.BackgroundAs of september 2021 there were 37M scientific articles in Wikidata, but 27M of them were missing any main
subject statement. That makes them very hard to find for scientists which is bad for science,
because building on the work of others is essential in the global scientific community.To my knowledge none of the scientific search engines that are currently used in the scientific community rely on an
open graph editable by anyone and maintained by the community itself for the purpose of helping fellow
scientists find each others work. Wikipedia and Scholia can fill that gap but we need good tooling to curate the
millions of items.CaveatThis type of matching that ONLY takes the label and not the underlying structured
data into account is SUBOPTIMAL. You are very welcome to suggest or contribute improvements
so we can improve the tool to help you make better edits.FeaturesThis tool has the following features:Adding a list of manually supplied main subjects to a few selected subgraphs
(These currently include a total of 37M items with scholarly items being the biggest subgraph by far).Matching against a set of items fetched via a SPARQL query.Matching up to a limit of items which together with Kubernetes makes it possible to start a query which
collects jobs with items until the limit is reached and then ask for approval/decline of each job. This
enables the user to create large batches of jobs with 100k+ items in total in a matter of minutes.Batch mode that can be used together with the above features and be run non-interactively
e.g. in the Wikimedia Cloud Services Kubernetes Beta clusterIt supportsWikidata:Edit groupsso that batches can easily be undone later if needed.
Click "details" in the summary of edits to see more.InstallationDownload the latest release with:$ pip install itemsubjectorAlternative installation in venvDownload the release tarball or clone the tool using Git.Clone the repositorygit clone https://github.com/dpriskorn/ItemSubjector.git && cd ItemSubjectorThen checkout the latest release.git checkout vx.x.xwhere x is the latest number on the release page.Setup the environmentMake a virtual environment and set it up using poetry. If you don't have poetry installed run:$ pip install poetryand then setup everying with$ poetry install --without=devto install all requirements in a virtual environment.PAWSNote: PAWS is not ideal for batch jobs unless you
are willing to keep your browser tab open for the
whole duration of the job. Consider using Kubernetes
instead, see belowThe tool runs in PAWS with no known
issues.log in to PAWSopen a terminalrungit clone https://github.com/dpriskorn/ItemSubjector.git .itemsubjector && cd .itemsubjector && pip install poetry && poetry install --without=dev<- note the dot in front of the directory name
that hides it from publication which is crucial to
avoid publication of your login credentials.follow the details under Setup belowWikimedia Cloud Services Kubernetes Beta clusterNote: this is for advanced users experienced with a SSH console environment, ask in theTelegram WikiCite groupif you need helpSeeKubernetes_HOWTO.mdSetupSetup the config by copying config/config.example.py ->
config/init.py and enter the botusername
(e.g. So9q@itemsubjector) and password
(firstcreate a botpasswordfor your account
and make sure you give it theedit page permissionandhigh volume permissions)e.g.cp config_example.py config.py && nano config.pyGNU Nano is an editor, pressctrl+xwhen you are done andyto save your changesUseThis tool helps by adding the
validated or supplied QID to all
scientific articles where the
search string appears (with
spaces around it or in the beginning
or end of the string) in the label
of the target item (e.g. scientific article).Adding QIDs manuallyAlways provide the most precise subjects firstRun the script with the -a or --add argument
followed by one or more QIDs or URLS:poetry run python itemsubjector.py -a Q108528107orpoetry run python itemsubjector.py -a https://www.wikidata.org/wiki/Q108528107Note since v0.2 you should not add subjects that are subclass
of each other in one go.
This is because of internal changes related to job handlingAdd the narrow first and then the broader like this:poetry run python itemsubjector.py -a narrow-QID && poetry run python itemsubjector.py -a broader-QIDPlease investigate before adding broad
subjects (with thousands of matches)
and try to nail down specific
subjects and add them first. If you are
unsure, please ask on-wiki or in theWikicite Telegram groupDisable alias matchingSometimes e.g. for main subjects likeSwedenit is necessary to disable alias matching to
avoid garbage matches.Usage example:poetry run python itemsubjector.py -a Q34 --no-aliases(the shorthand-naalso works)Disable search expression confirmationAvoid the extra question "Do you want to continue?":Usage example:poetry run python itemsubjector.py -a Q34 --no-confirmation(the shorthand-ncalso works)Show links column in table of search expressionsThis is handy if you want to look them up easily.Usage example:poetry run python itemsubjector.py -a Q34 --show-search-urls(the shorthand-sualso works)Show links column in table of search expressionsThis is handy if you want to look them up easily.Usage example:poetry run python itemsubjector.py -a Q34 --show-item-urls(the shorthand-iualso works)Matching main subjects based on a SPARQL query.The tool can create a list of jobs by picking random subjects from a
users SPARQL query.Usage example for diseases:poetry run python itemsubjector.py -iu --sparql "SELECT ?item WHERE {?item wdt:P31 wd:Q12136.}"This makes it much easier to cover a range a subjects.
This example query returns ~5000 items to match :)Batch job featuresThe tool can help prepare jobs and then run
them later non-interactively. This enables the user
to submit them as jobs on the Wikimedia Cloud Service
Beta Kubernetes cluster, so you don't
have to run them locally if you don't want to.See the commands below andhttps://phabricator.wikimedia.org/T285944#7373913for details.Note: if you quit/stop a list of jobs that are
currently running, please remove the
unfinished prepared jobs before preparing
new jobs by running --remove-prepared-jobsList of all optionsThis is the output ofitemsubjector.py -h:usage: itemsubjector.py [-h] [-a ADD [ADD ...]] [-na] [-nc] [-p] [-r] [-rm] [-m] [-w] [-su] [-iu] [--sparql [SPARQL]] [--debug-sparql]
                        [--no-ask-match-more-limit [NO_ASK_MATCH_MORE_LIMIT]] [--export-jobs-to-dataframe]

ItemSubjector enables working main subject statements on items based on a
heuristic matching the subject with the title of the item.

Example adding one QID:
'$ itemsubjector.py -a Q1234'

Example adding one QID and prepare a job list to be run non-interactively later:
'$ itemsubjector.py -a Q1234 -p'

Example working on all diseases:
'$ itemsubjector.py --sparql "SELECT ?item WHERE {?item wdt:P31 wd:Q12136. MINUS {?item wdt:P1889 [].}}"'


optional arguments:
  -h, --help            show this help message and exit
  -a ADD [ADD ...], --add ADD [ADD ...], --qid-to-add ADD [ADD ...]
                        List of QIDs or URLs to Q-items that are to be added as main subjects on scientific articles. Always add the most specific ones first. See the
                        README for examples
  -na, --no-aliases     Turn off alias matching
  -nc, --no-confirmation
                        Turn off confirmation after displaying the search expressions, before running the queries.
  -p, --prepare-jobs    Prepare a job for later execution, e.g. in a job engine
  -r, --run-prepared-jobs
                        Run prepared jobs non-interactively
  -rm, --remove-prepared-jobs
                        Remove prepared jobs
  -m, --match-existing-main-subjects
                        Match from list of 136.000 already used main subjects on other scientific articles
  -su, --show-search-urls
                        Show an extra column in the table of search strings with links
  -iu, --show-item-urls
                        Show an extra column in the table of items with links
  --sparql [SPARQL]     Work on main subject items returned by this SPARQL query. Note: "?item" has to be selected for it to work, see the example above. Note: MINUS {?item
                        wdt:P1889 [].} must be present in the query to avoid false positives.
  --debug-sparql        Enable debugging of SPARQL queries.
  --no-ask-match-more-limit [NO_ASK_MATCH_MORE_LIMIT], --limit [NO_ASK_MATCH_MORE_LIMIT]
                        When working on SPARQL queries of e.g. galaxies, match more until this many matches are in the job list
  --export-jobs-to-dataframe
                        Export the prepared job list to a Pandas DataFrame.What I learnedI used the black code-formatter for the first time in this project and
it is a pleasure to not have to sit and manually format the code anymore.I used argparse for the first time in this project and how to type it
properly.This was one of the first of my projects that had scope creep. I have
removed the QuickStatements export to simplify the program.This project has been used in a scientific paper I wrote together withHoucemeddine TurkiRewrite 2022:Important to break down methods to 1 method 1 task to increase readability. -> helps reuse in other projects.Important to avoid resetting attributes and instantiate classes instead. -> helps reuse in other projects.Simplify as much as possible to keep the whole thing lean and avoid scope creep. -> helps reuse in other projects. (KISS-principle)Difficult to judge which features are used and which are not. User testing would be nice.UML diagrams are nice. They give a good quick overview.Removing options that no-one seems to use helps keeping it simple. It would be valuable to get better insight of how the
program is used by the users. A discussion in github might help in this.ThanksDuring the development of this tool the author got a
help multiple times fromJan AinaliandJon Søbywith figuring out how to query the API using the
CirrusSearch extensions and to remove more
specific main subjects from the query results.A special thanks also toMagnus SälgöandArthur Smithfor their
valuable input and ideas, e.g. to search for aliases also and toJeanand the
incredibly
helpful people in the Wikimedia Cloud Services Support chat that
helped with making batch jobs run successfully.Thanks also tojsamwritesfor help with testing and suggestions
for improvement and for using the tool to improve a ton of items :).LicenseGPLv3+