Repository: dpriskorn/ItemSubjector (language: python)

Issue:
Support dissertations also
see https://query.wikidata.org/#%23Katter%0ASELECT%20%28count%28%3Fitem%29%20as%20%3Fc%29%20%0AWHERE%20%0A%7B%0A%20%20%3Fitem%20wdt%3AP31%2Fwdt%3AP279%2a%20wd%3AQ1266946.%20%23%20M%C3%A5ste%20vara%20av%20en%20katt%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%2Cen%22.%20%7D%20%23%20%3Cspan%20lang%3D%22en%22%20dir%3D%22ltr%22%20class%3D%22mw-content-ltr%22%3EHelps%20get%20the%20label%20in%20your%20language%2C%20if%20not%2C%20then%20en%20language%3C%2Fspan%3E%0A%7D where 52000 is found
Increase sample size with batch size
A rule of thumb could be for every 20 items - show a sample, but always a minimum of 50.
That means:
500 -> 50
1000 -> 50
2000 -> 100
3000 -> 150
4000 -> 200
5000 -> 250
6000 -> 300

for batches larger than 4000 throw a warning that the batch size is so big that you should split it up by first running --no-aliases if possible



Discussion / hints:

Fix patch:
diff --git a/.gitignore b/.gitignore
index f85c6b1..f787f66 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,1 +1,2 @@
-config.py
\ No newline at end of file
+config.py
+pickle.dat
\ No newline at end of file
diff --git a/Kubernetes_HOWTO.md b/Kubernetes_HOWTO.md
new file mode 100644
index 0000000..71353df
--- /dev/null
+++ b/Kubernetes_HOWTO.md
@@ -0,0 +1,48 @@
+# How to setup ItemSubjector on Kubernetes
+So you have access to a cluster and you want to run ItemSubjector in batch mode?
+
+The guide below is adapted to the specifics of the Wikimedia cluster, 
+but it should be possible to run on any Kubernetes cluster.
+
+## Wikimedia Cloud VCS Kubernetes setup
+1) setup a toolforge account, see https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Quickstart
+
+## Windows
+2) download PuTTY (e.g. with chocolatey.org)
+3) [generate a SSH key using PuTTY](https://phoenixnap.com/kb/generate-ssh-key-windows-10) and give it a password as per WMF instructions here https://www.mediawiki.org/wiki/Gerrit/Tutorial#Generate_a_new_SSH_key
+4) log in via PuTTY to url `dev-buster.toolforge.org` on port 22, see https://www.mediawiki.org/wiki/Toolserver:Logging_in#Logging_in_with_PuTTY. Ask for help in [Telegram](https://t.me/wmcloudirc) or IRC if you don't succeed
+
+## Create a tool
+1) Now log into the [toolserver webinterface](https://toolsadmin.wikimedia.org/tools/) and [create a tool](https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Tool_Accounts#Create_tools_with_Tool_Accounts). E.g. "itemsubjector-YOUR_USERNAME"
+
+## Become tool
+After the tool is registered log out of SSH and back in.
+
+1) become the tool `become TOOLNAME`
+
+## GNU screen
+The author recommends GNU screen to make it possible 
+to have multiple "windows" and to be able to easily attach/detach
+
+If you don't know how to use screen your life can become pretty miserable. Read up and watch e.g. https://www.youtube.com/results?search_query=gnu+screen
+
+The author recommends:
+- increasing the scrollback buffer by invoking with e.g. `screen -D -RR -h 5000`
+- using `ctrl + a ESC` to scroll back and inspect matches
+
+## Clone ItemSubjector
+run `git clone https://github.com/dpriskorn/ItemSubjector.git itemsubjector && cd itemsubjector`
+
+## Fix the execution right on the scripts
+Run this in the itemsubjector-folder: 
+`chmod +x *.sh`
+
+## Prepare a set of jobs
+Follow the README, e.g. run `python itemsubjector.py -a Q108801503`
+
+## Start a batch on Kubernetes
+run `./create_kubernettes_job_and_watch_the_log.sh 1`
+
+This will start he k8s job and show you the tail of the output. 
+By using `watch` the output will be updated every 2 seconds 
+which makes it easy for you to get an idea of the progress.
\ No newline at end of file
diff --git a/README.md b/README.md
index 0e2b6c1..4e8e8cc 100644
--- a/README.md
+++ b/README.md
@@ -1,23 +1,24 @@
 # ItemSubjector
-*still in BETA*
+*This is beta software, expect new features, renamed arguments, bugs, etc.*
 
 Tool made to add main subject statements to 
-items based on the title using a home-brewed 
-CirrusSearch-based Named Entity Recognition algorithm. 
+items based on a heuristic matching the subject with the title of the item. 
 ![bild](https://user-images.githubusercontent.com/68460690/133230724-40a610b7-5557-4b2b-b66e-2d80ca89e90d.png)
 *The tool running in PAWS adding manually found main subject QIDs*
 
 # Features
-This tool has the following features
-* adding a list of main subjects to items 
-  (for now only scholarly articles are supported)
-* automatically extracting n-grams from labels of 10.000 articles 
-  (this is not that powerful because users know better than scikit 
-  what subjects are meaningful to have on our scientific articles)
+This tool has the following features:
+* Adding a list of manually supplied main subjects to a few selected subgraphs 
+  (These currently include a total of 37M items with scholarly items being the biggest subgraph by far).
+* Matching against a set of items fetched via a SPARQL query.
+* Matching against a downloaded set of existing main subjects and adding them to more scholarly articles
+* Batch mode that can be used together with the above features and be run non-interactively 
+  e.g. in the Wikimedia Cloud Services Kubernetes Beta cluster
 
 It supports 
 [Wikidata:Edit groups](https://www.wikidata.org/wiki/Wikidata:Edit_groups) 
-so that batches can easily be undone if needed.
+so that batches can easily be undone later if needed. 
+Click "details" in the summary of edits to see more.
 
 # Thanks
 During the development of this tool the author got a 
@@ -27,10 +28,16 @@ CirrusSearch extensions and to remove more
 specific main subjects from the query results.
 
 A special thanks also to **Magnus Sälgö** for his valuable input 
-and ideas, e.g. to search for aliases also.
+and ideas, e.g. to search for aliases also and to *Jean* and the 
+incredibly
+helpful people in the Wikimedia Cloud Services Support chat that
+helped with making batch jobs run successfully.
 
 # Installation
-Clone the repo and run
+Clone the repo 
+`git clone https://github.com/dpriskorn/ItemSubjector.git && cd ItemSubjector`
+
+and run
 
 `pip install -r requirements.txt`
 
@@ -41,10 +48,19 @@ The tool runs in PAWS with no known
 issues.
 * log in to PAWS
 * open a terminal
-* run `git clone https://github.com/dpriskorn/ItemSubjector.git .itemsubjector && cd .itemsubjector` 
+* run `git clone https://github.com/dpriskorn/ItemSubjector.git .itemsubjector && cd .itemsubjector && pip install -r requirements.txt` 
   <- note the dot in front of the directory name 
-  that hides it from publication.
-* run the pip-command above
+  that hides it from publication which is crucial to 
+  avoid publication of your login credentials.
+* follow the details under Setup below
+
+*Note: PAWS is not ideal for batch jobs unless you 
+are willing to keep your browser tab open for the 
+whole duration of the job. Consider using Kubernetes 
+instead, see below*
+
+## Wikimedia Cloud Services Kubernetes Beta cluster
+See [Kubernetes_HOWTO.md](Kubernetes_HOWTO.md)
 
 # Setup
 Like my other tools, copy config.example.py -> 
@@ -52,18 +68,15 @@ config.py and enter the botusername
 (e.g. So9q@itemsubjector) and password
 * e.g. `cp config.example.py config.py && nano config.py`
 
-# Use
-It has 2 modes:
-1) automatic finding n-grams and trying to 
-   detect items that match (default if no
-   arguments are given on the command line)
-2) add main subject items to scholarly articles (see details below)
+*GNU Nano is an editor, press `ctrl+x` when you are done and `y` to save your changes*
 
-Both modes conclude by adding the 
+# Use
+This tool helps by adding the 
 validated or supplied QID to all 
 scientific articles where the 
-n-gram or search string appears (with 
-spaces around it) in the label 
+search string appears (with 
+spaces around it or in the beginning
+or end of the string) in the label 
 of the target item (e.g. scientific article).
 
 ## Adding QIDs manually
@@ -71,25 +84,15 @@ of the target item (e.g. scientific article).
 
 Run the script with the -l or --list argument 
 followed by one or more QIDs or URLS:
-* `python itemsubjector.py -l Q108528107` or
-* `python itemsubjector.py -l https://www.wikidata.org/wiki/Q108528107`
-  
-Here is a more advanced example:
-The first is *metastatic breast cancer* which is a 
-subclass of the second *breast cancer*
-* `python itemsubjector.py -l Q108528107 Q128581` or
-
-In this case the tool is smart enough 
-(thanks to Jan Ainali) to first add 
-*metastatic breast cancer* to items 
-and then exclude those items when adding the 
-more general subject afterwards.
-
-This way we avoid redundancy since we 
-want the most specific subjects on the 
-items and not 
-all the general ones above it in the 
-classification system.
+* `python itemsubjector.py -a Q108528107` or
+* `python itemsubjector.py -a https://www.wikidata.org/wiki/Q108528107`
+
+*Note since v0.2 you should not add subjects that are subclass 
+of each other in one go. 
+This is because of internal changes related to job handling*
+
+Add the narrow first and then the broader like this:
+* `python itemsubjector.py -a narrow-QID && python itemsubjector.py -l broader-QID`
 
 Please investigate before adding broad 
 subjects (with thousands of matches) 
@@ -105,24 +108,131 @@ it is necessary to disable alias matching to
 avoid garbage matches. 
 
 Usage example:
-`python itemsubjector.py -l Q34 --no-aliases` 
+`python itemsubjector.py -a Q34 --no-aliases` 
 (the shorthand `-na` also works)
 
+### Disable search expression confirmation
+Avoid the extra question "Do you want to continue?":
+
+Usage example:
+`python itemsubjector.py -a Q34 --no-confirmation` 
+(the shorthand `-nc` also works)
+
+### Show links column in table of search expressions 
+This is handy if you want to look them up easily.
+
+Usage example:
+`python itemsubjector.py -a Q34 --show-search-urls` 
+(the shorthand `-su` also works)
+
+### Show links column in table of search expressions 
+This is handy if you want to look them up easily.
+
+Usage example:
+`python itemsubjector.py -a Q34 --show-item-urls` 
+(the shorthand `-iu` also works)
+
+### Limit to scholarly articles without main subject
+Usage example:
+`python itemsubjector.py -l Q34 --limit-to-items-without-p921` 
+(the shorthand `-w` also works)
+
+## Matching main subjects based on a SPARQL query.
+The tool can create a list of jobs by picking random subjects from a
+users SPARQL query.
+
+Usage example for diseases:
+`python itemsubjector.py -iu --sparql "SELECT ?item WHERE {?item wdt:P31 wd:Q12136.}"`
+
+This makes it much easier to cover a range a subjects. 
+This example query returns ~5000 items to match :)
+
+## Matching against existing main subjects
+*Warning: This often proposes too broad subjects 
+that should most probably not be added as main subjects to all matches, 
+but instead be broken down into narrower subjects*
+
+The tool can create a list of jobs by picking random subjects from a
+big list fetched from WDQS.
+
+This enables the user to quickly build up a big list of jobs to run 
+and improve the graph by improving the coverage on existing subjects.
+
+To set it up run:
+* `python fetch_main_subjects.py` by default it fetches 100,000 main 
+  subjects and saves them after removing duplicates 
+  (this usually results in about 10,000 unique subjects which should 
+  be enough to get started)
+  The script implements a random offset so running it again will yield 
+  a different set of subjects, but please don't run it too often.
+
+Usage example:
+* `python itemsubjector.py -m`
+
+## Batch job features
+The tool can help prepare jobs and then run 
+them later non-interactively. This enables the user
+to submit them as jobs on the Wikimedia Cloud Service 
+Beta Kubernetes cluster, so you don't 
+have to run them locally if you don't want to.
+
+See the commands below and 
+https://phabricator.wikimedia.org/T285944#7373913 
+for details.
+
+*Note: if you quit/stop a list of jobs that are 
+currently running, please remove the 
+unfinished prepared jobs before preparing 
+new jobs by running --remove-prepared-jobs*
+
 ## List of all options
 This is the output of `itemsubjector.py -h`:
 ```buildoutcfg
-usage: itemsubjector.py [-h] [-l LIST [LIST ...]] [-na]
+usage: itemsubjector.py [-h] [-a ADD [ADD ...]] [-na] [-nc] [-p] [-r] [-rm] [-m] [-w] [-su] [-iu] [--sparql [SPARQL]] [--debug-sparql]
+
+ItemSubjector enables working main subject statements on items based on a
+heuristic matching the subject with the title of the item.
+
+Example adding one QID:
+'$ itemsubjector.py -l Q1234'
+
+Example adding one QID and prepare a job list to be run non-interactively later:
+'$ itemsubjector.py -l Q1234 -p'
+
+Example adding random QIDs from a list of main subjects extracted from 2 million scholarly articles:
+'$ itemsubjector.py -m'
+
+Example adding random QIDs from a list of main subjects extracted from 2 million scholarly articles
+and prepare a job list:
+'$ itemsubjector.py -m -p'
+
+Example working on all diseases:
+'$ itemsubjector.py --sparql "SELECT ?item WHERE {?item wdt:P31 wd:Q12136.}"'
+
 
 optional arguments:
   -h, --help            show this help message and exit
-  -l LIST [LIST ...], --list LIST [LIST ...]
-                        List of QIDs or URLs to Q-items 
-                        that are to be added as main subjects 
-                        on scientific articles. Always add the 
-                        most specific ones first. 
-                        See the README for examples
+  -a ADD [ADD ...], --add ADD [ADD ...], --qid-to-add ADD [ADD ...]
+                        List of QIDs or URLs to Q-items that are to be added as main subjects on scientific articles. Always add the most specific ones first. See the README
+                        for examples
   -na, --no-aliases     Turn off alias matching
-
+  -nc, --no-confirmation
+                        Turn off confirmation after displaying the search expressions, before running the queries.
+  -p, --prepare-jobs    Prepare a job for later execution, e.g. in a job engine
+  -r, --run-prepared-jobs
+                        Run prepared jobs non-interactively
+  -rm, --remove-prepared-jobs
+                        Remove prepared jobs
+  -m, --match-existing-main-subjects
+                        Match from list of 136.000 already used main subjects on other scientific articles
+  -w, --limit-to-items-without-p921
+                        Limit matching to scientific articles without P921 main subject
+  -su, --show-search-urls
+                        Show an extra column in the table of search strings with links
+  -iu, --show-item-urls
+                        Show an extra column in the table of items with links
+  --sparql [SPARQL]     Work on main subject items returned by this SPARQL query. Note: "?item" has to be in selected for it to work.
+  --debug-sparql        Enable debugging of SPARQL queries.
 ```
 # License
 GPLv3+
diff --git a/config.example.py b/config.example.py
index 004e865..a3dc164 100644
--- a/config.example.py
+++ b/config.example.py
@@ -1,11 +1,12 @@
-import os
+import tempfile
+from pathlib import Path
 
 # Add your botpassword and login here:
 username = ""
 password = ""
 
 # Global settings
-version = "0.1"  # Don't touch this.
+version = "0.2-beta2"  # Don't touch this.
 wd_prefix = "http://www.wikidata.org/entity/"
 endpoint = "https://query.wikidata.org/sparql"
 wiki_user = "User:So9q"  # Change this to your username
@@ -13,4 +14,7 @@
 tool_url = "https://github.com/dpriskorn/ItemSubjector"
 tool_wikipage = "Wikidata:Tools/ItemSubjector"
 login_instance = None
-random_offset=0
\ No newline at end of file
+# This should work for all platforms except kubernetes
+job_pickle_file_path = f"{tempfile.gettempdir()}/pickle.dat"
+# job_pickle_file_path = "~/pickle.dat"  # works on kubernetes
+main_subjects_pickle_file_path = f"data/main_subjects.pkl"
diff --git a/create_kubernettes_job_and_watch_the_log.sh b/create_kubernettes_job_and_watch_the_log.sh
new file mode 100644
index 0000000..4c417c7
--- /dev/null
+++ b/create_kubernettes_job_and_watch_the_log.sh
@@ -0,0 +1,4 @@
+# Usage: chmod +x this script and run like this "./create_kubernettes_job_and_watch_the_log.sh 1"
+# Increment the job number yourself each time
+toolforge-jobs run job$1 --image tf-python39 --command "~/setup.sh && python3 ~/itemsubjector/itemsubjector.py -r"
+watch tail ~/job$1*
diff --git a/data/main_subjects.sparql b/data/main_subjects.sparql
new file mode 100644
index 0000000..dd8e1b8
--- /dev/null
+++ b/data/main_subjects.sparql
@@ -0,0 +1,27 @@
+#%endpoint https://query.wikidata.org/sparql
+#title: Get main subjects used at least once on scholarly articles
+SELECT ?subject 
+WHERE 
+{
+    {
+        SELECT * WHERE {
+            hint:Query hint:optimizer "None".
+            ?item wdt:P31 wd:Q13442814;
+                  wdt:P921 ?subject.
+        } 
+        #offset 1000000
+        limit 100000
+    }
+  MINUS{
+      ?item wdt:P31 wd:Q8054.  # protein
+  }
+  MINUS{
+      ?item wdt:P279 wd:Q8054.  # protein
+  }
+  MINUS{
+      ?item wdt:P31 wd:Q7187.  # gene
+  }
+  MINUS{
+      ?item wdt:P279 wd:Q7187.  # gene
+  }
+}
diff --git a/fetch_main_subjects.py b/fetch_main_subjects.py
new file mode 100644
index 0000000..27e83cd
--- /dev/null
+++ b/fetch_main_subjects.py
@@ -0,0 +1,71 @@
+import logging
+import random
+
+from wikibaseintegrator import wbi_config
+from wikibaseintegrator.wbi_helpers import execute_sparql_query
+
+import config
+from src import console
+from src.helpers.cleaning import strip_prefix
+from src.helpers.pickle import add_to_main_subject_pickle
+
+logging.basicConfig(level=logging.DEBUG)
+logger = logging.getLogger(__name__)
+wbi_config.config["USER_AGENT_DEFAULT"] = config.user_agent
+console.print("Fetching 100,000 main subjects")
+console.input("Press enter to continue")
+subjects = []
+# This offset ensures that we don't get
+# the same subset of subjects every time we run it
+randomizing_offset: int = random.randint(1, 500000)
+console.print(f"Random offset used: {randomizing_offset} for this run")
+for i in range(0+randomizing_offset, 100000+randomizing_offset, 10000):
+    print(i)
+    # title: Get main subjects used at least once on scholarly articles
+    results = execute_sparql_query(f"""
+SELECT ?subject
+WHERE
+{{
+{{
+SELECT DISTINCT ?subject WHERE {{
+    hint:Query hint:optimizer "None".
+    ?item wdt:P31 wd:Q13442814;
+          wdt:P921 ?subject.
+}}
+offset {i}
+limit 10000
+}}
+MINUS{{
+?item wdt:P31 wd:Q8054.  # protein
+}}
+MINUS{{
+?item wdt:P279 wd:Q8054.  # protein
+}}
+MINUS{{
+?item wdt:P31 wd:Q7187.  # gene
+}}
+MINUS{{
+?item wdt:P279 wd:Q7187.  # gene
+}}
+}}
+    """)
+    if len(results) == 0:
+        raise ValueError("No main subjects found")
+    else:
+        # print("adding lexemes to list")
+        # pprint(results.keys())
+        # pprint(results["results"].keys())
+        # pprint(len(results["results"]["bindings"]))
+        for result in results["results"]["bindings"]:
+            # print(result)
+            subjects.append(strip_prefix(result["subject"]["value"]))
+            # exit(0)
+console.print(f"{len(subjects)} fetched")
+console.print("Filtering out duplicates")
+subjects_without_duplicates = set()
+for subject in subjects:
+    subjects_without_duplicates.add(subject)
+console.print(f"Saving {len(subjects_without_duplicates)} "
+              f"to pickle '{config.main_subjects_pickle_file_path}' (overwriting)")
+add_to_main_subject_pickle(subjects)
+console.print("Done")
\ No newline at end of file
diff --git a/helpers/console.py b/helpers/console.py
deleted file mode 100644
index 8aea885..0000000
--- a/helpers/console.py
+++ /dev/null
@@ -1,102 +0,0 @@
-from typing import List, Dict
-
-from rich.console import Console
-from rich.table import Table
-
-from models.wikidata import Items
-
-console = Console()
-
-
-def ask_yes_no_question(message: str):
-    # https://www.quora.com/
-    # I%E2%80%99m-new-to-Python-how-can-I-write-a-yes-no-question
-    # this will loop forever
-    while True:
-        answer = console.input(message + ' [Y/Enter/n]: ')
-        if len(answer) == 0 or answer[0].lower() in ('y', 'n'):
-            if len(answer) == 0:
-                return True
-            else:
-                # the == operator just returns a boolean,
-                return answer[0].lower() == 'y'
-
-
-def introduction():
-    print_keep_an_eye_on_wdqs_lag()
-    console.input(
-        "This tool enables you to find n-grams from labels "
-        "semi-automatically and validate the match between the n-grams "
-        "with items found by searching Wikidata.\n"
-        "E.g. the 2-gram 'breast cancer' corresponds to the item: Q128581: "
-        "Breast cancer: cancer that originates in the mammary gland.\n"
-        "The tool makes it simple to add main subject to a lot of items "
-        "(in the example above there are ~8000 matches).\n"
-        "Note: If unsure you should reject a match when validating.\n"
-        "Press Enter to start."
-    )
-
-
-def print_keep_an_eye_on_wdqs_lag():
-    console.print("Please keep an eye on the lag of the WDQS cluster here and avoid "
-                  "working if it is over a few minutes.\n"
-                  "https://grafana.wikimedia.org/d/000000489/wikidata-query-service?"
-                  "orgId=1&viewPanel=8&from=now-30m&to=now&refresh=1d")
-
-
-def press_enter_to_start():
-    console.input("Press Enter to start.")
-
-
-def print_scholarly_articles_best_practice_information():
-    print_keep_an_eye_on_wdqs_lag()
-    console.print(
-        "When adding QID main subjects please try to first "
-        "educate yourself about the subarea of science a little "
-        "and find/create items as specific as possible.\n"
-        "E.g. when searching for 'cancer screening' in Wikidata "
-        "we find 'gastric cancer screening' in labels of "
-        "scientific articles but there is "
-        "perhaps no item for this yet.\n"
-        "In this case it is preferred to first create that item "
-        "(done in Q108532542 and add that as main subject and "
-        "avoid the more general 'cancer screening' until all "
-        "subforms of screening are added."
-    )
-    press_enter_to_start()
-
-
-def print_riksdagen_documents_best_practice_information():
-    print_keep_an_eye_on_wdqs_lag()
-    press_enter_to_start()
-
-
-def print_ngram_table(results: Dict):
-    table = Table(title="N-grams found")
-    table.add_column("N-gram")
-    table.add_column("Frequency")
-    for ngram in results:
-        table.add_row(ngram, str(results[ngram]))
-    console.print(table)
-
-
-def print_search_strings_table(search_strings: List[str]):
-    table = Table(title="Search strings")
-    table.add_column(f"Extracted the following {len(search_strings)} search strings:")
-    for string in search_strings:
-        table.add_row(string)
-    console.print(table)
-
-
-def print_found_items_table(items: Items = None):
-    if items is None:
-        raise ValueError("items was None")
-    table = Table(title="Matched items found")
-    table.add_column("Showing only the first 50 items if more are found")
-    for item in items.list[0:50]:
-        table.add_row(item.label)
-    console.print(table)
-
-
-def ask_continue_with_the_rest():
-    console.input("Press enter to continue or ctrl+c to quit")
diff --git a/helpers/sparql_dataframe.py b/helpers/sparql_dataframe.py
deleted file mode 100644
index 7bf3e4c..0000000
--- a/helpers/sparql_dataframe.py
+++ /dev/null
@@ -1,21 +0,0 @@
-import pandas as pd
-from SPARQLWrapper import SPARQLWrapper, JSON
-
-
-def query_wikidata(sparql_query, sparql_service_url):
-    """Query the endpoint with the given query string
-    and return the results as a pandas Dataframe."""
-    # create the connection to the endpoint
-    # Wikidata enforces now a strict User-Agent policy, we need to specify the agent
-    # See here https://www.wikidata.org/wiki/Wikidata:Project_chat/Archive/2019/07#problems_with_query_API
-    # https://meta.wikimedia.org/wiki/User-Agent_policy
-    sparql = SPARQLWrapper(sparql_service_url, agent="Sparql Wrapper on PAWS, User:So9q")
-
-    sparql.setQuery(sparql_query)
-    sparql.setReturnFormat(JSON)
-
-    # ask for the result
-    from helpers.console import console
-    console.status("Fetching data from WDQS...")
-    result = sparql.query().convert()
-    return pd.json_normalize(result["results"]["bindings"])
diff --git a/itemsubjector.py b/itemsubjector.py
index 8f56b58..344ed19 100644
--- a/itemsubjector.py
+++ b/itemsubjector.py
@@ -1,208 +1,6 @@
-import argparse
 import logging
 
-from wikibaseintegrator import wbi_login, wbi_config
-from wikibaseintegrator.datatypes import Item as ItemType
+from src import *
 
-import config
-from helpers.calculations import calculate_random_editgroups_hash
-from helpers.console import console, ask_yes_no_question, introduction, print_ngram_table, \
-    print_scholarly_articles_best_practice_information, print_riksdagen_documents_best_practice_information, \
-    print_found_items_table, ask_continue_with_the_rest
-from helpers.enums import TaskIds
-from helpers.menus import select_task
-from models.ngram import NGram
-from models.riksdagen_documents import RiksdagenDocumentItems
-from models.scholarly_articles import ScholarlyArticleItems
-from models.suggestion import Suggestion
-from models.task import Task
-from models.wikidata import Item
-from tasks import tasks
-
-logging.basicConfig(level=logging.WARNING)
-
-# pseudo code
-# let user choose what to work on ie.
-# let user choose language and subgraph
-# e.g. Swedish documents from Riksdagen
-# e.g. English scientific articles
-
-# loop:
-# get some labels
-# let the user chose 1 meaningful match from our home cooked NER
-# download all items without main subject matching the matched entity
-# and with the entity label in the item label
-# upload main subject to all
-
-
-def add_suggestion_to_items(suggestion: Suggestion = None,
-                            task: Task = None):
-    """Add a suggested QID as main subject on all items that
-    have a label that matches one of the search strings for this QID
-    We calculate a new edit group hash each time this function is
-    called so similar edits are grouped and easily be undone."""
-    if suggestion is None:
-        raise ValueError("suggestion was None")
-    if task is None:
-        raise ValueError("task was None")
-    with console.status(f'Fetching items with labels that have one of '
-                        f'the search strings by running a total of '
-                        f'{len(suggestion.search_strings)} queries on WDQS...'):
-        if task.id == TaskIds.SCHOLARLY_ARTICLES:
-            items = ScholarlyArticleItems()
-        elif task.id == TaskIds.RIKSDAGEN_DOCUMENTS:
-            items = RiksdagenDocumentItems()
-        else:
-            raise ValueError(f"{task.id} was not recognized")
-        items.fetch_based_on_label(suggestion=suggestion,
-                                   task=task)
-    if len(items.list) > 0:
-        # Randomize the list
-        items.random_shuffle_list()
-        print_found_items_table(items=items)
-        ask_continue_with_the_rest()
-        editgroups_hash: str = calculate_random_editgroups_hash()
-        count = 0
-        for item in items.list:
-            count += 1
-            with console.status(f"Uploading main subject [green]{suggestion.item.label}[/green] to {item.label}"):
-                main_subject_property = "P921"
-                reference = ItemType(
-                    "Q69652283",  # inferred from title
-                    prop_nr="P887"  # based on heuristic
-                )
-                statement = ItemType(
-                    suggestion.item.id,
-                    prop_nr=main_subject_property,
-                    references=[reference]
-                )
-                item.upload_one_statement_to_wikidata(
-                    statement=statement,
-                    summary=f"[[Property:{main_subject_property}]]: [[{suggestion.item.id}]]",
-                    editgroups_hash=editgroups_hash
-                )
-            console.print(f"({count}/{len(items.list)}) "
-                          f"Added '{suggestion.item.label}' to {item.label}: {item.url()}")
-            # input("Press enter to continue")
-    else:
-        console.print("No matching items found")
-
-
-def process_ngrams(results):
-    suggestions = []
-    with console.status("Searching the Wikidata API for entities matching the found n-grams..."):
-        for result in results:
-            ngram = NGram(
-                label=result,
-                frequency=results[result]
-            )
-            suggestion = ngram.recognize_named_entity()
-            if suggestion is not None:
-                suggestions.append(suggestion)
-    console.print("[bold,green]Found the following candidates:")
-    for suggestion in suggestions:
-        answer = ask_yes_no_question(f"{str(suggestion)}\n"
-                                     f"Is this a valid main subject?")
-        if answer:
-            add_suggestion_to_items(suggestion=suggestion)
-        else:
-            console.print("Skipping this suggestion")
-        console.print("\n")
-
-
-def process_user_supplied_qids(args=None, task: Task = None):
-    """Given a list of QIDs, we go through
-    them and call add_suggestion_to_items() on each one"""
-    logger = logging.getLogger(__name__)
-    if args is None:
-        raise ValueError("args was None")
-    if task is None:
-        raise ValueError("task was None")
-    if task.id == TaskIds.SCHOLARLY_ARTICLES:
-        print_scholarly_articles_best_practice_information()
-    elif task.id == TaskIds.RIKSDAGEN_DOCUMENTS:
-        print_riksdagen_documents_best_practice_information()
-    else:
-        raise ValueError(f"taskid {task.id} not recognized")
-    login()
-    for qid in args.list:
-        if "https://www.wikidata.org/wiki/" in qid:
-            qid = qid[30:]
-        if "http://www.wikidata.org/entity/" in qid:
-            qid = qid[31:]
-        logger.debug(f"qid:{qid}")
-        item = Item(
-            id=qid,
-            task=task
-        )
-        console.print(f"Working on {item}")
-        # generate suggestion with all we need
-        suggestion = Suggestion(
-            item=item,
-            ngram=NGram(
-                label=item.label,
-                frequency=None
-            ),
-            task=task,
-            args=args
-        )
-        add_suggestion_to_items(suggestion=suggestion,
-                                task=task)
-
-
-def login():
-    with console.status("Logging in with WikibaseIntegrator..."):
-        config.login_instance = wbi_login.Login(
-            auth_method='login',
-            user=config.username,
-            password=config.password,
-            debug=False
-        )
-        # Set User-Agent
-        wbi_config.config["USER_AGENT_DEFAULT"] = config.user_agent
-
-
-def main():
-    # logger = logging.getLogger(__name__)
-    parser = argparse.ArgumentParser()
-    # TODO support turning off aliases
-    parser.add_argument('-l', '--list',
-                        nargs='+',
-                        help=('List of QIDs or URLs to Q-items that '
-                              'are to be added as '
-                              'main subjects on scientific articles. '
-                              'Always add the most specific ones first. '
-                              'See the README for examples'),
-                        required=False)
-    parser.add_argument('-na', '--no-aliases',
-                        action='store_true',
-                        help='Turn off alias matching'
-                        )
-    args = parser.parse_args()
-    # console.print(args.list)
-    if args.list is None:
-        introduction()
-        login()
-        # disabled for now
-        # select_language()
-        # task: Task = select_task()
-        # if task is None:
-        #     raise ValueError("Got no task")
-        # We only have 1 task so don't bother about showing the menu
-        task = tasks[0]
-        ngrams: dict = task.labels.get_ngrams()
-        # logger.debug(results)
-        if ngrams is not None:
-            print_ngram_table(ngrams)
-            process_ngrams(ngrams)
-        else:
-            raise ValueError("results was None")
-    else:
-        task: Task = select_task()
-        if task is None:
-            raise ValueError("Got no task")
-        process_user_supplied_qids(args=args, task=task)
-
-
-if __name__ == "__main__":
-    main()
+logging.basicConfig(level=logging.DEBUG)
+main()
\ No newline at end of file
diff --git a/models/ngram.py b/models/ngram.py
deleted file mode 100644
index 5f85e58..0000000
--- a/models/ngram.py
+++ /dev/null
@@ -1,76 +0,0 @@
-import logging
-
-import requests
-from wikibaseintegrator.wbi_helpers import search_entities
-
-import config
-from models.wikidata import Item
-
-
-class NGram:
-    label: str
-    frequency: int
-
-    def __init__(self,
-                 label: str = None,
-                 frequency: int = None):
-        self.label = label
-        self.frequency = frequency
-
-    def recognize_named_entity(self):  # typing: -> Suggestion
-        if self.label is None:
-            raise ValueError("ngram was None")
-        # console.print(f"Results for {self.label}")
-        search_expression = (
-                "-haswbstatement:P31=Q13442814 " +  # scientific article
-                "-haswbstatement:P31=Q5633421 " +  # journal
-                # See
-                f'"{self.label}"'
-        )
-        params = (
-            ('format', 'json'),
-            ('action', 'query'),
-            ('list', 'search'),
-            ('srprop', ''),
-            ('srlimit', '10'),
-            ('sroffset', '0'),
-            ('srsearch', search_expression),
-        )
-        headers = {
-            'User-Agent': config.user_agent,
-        }
-        response = requests.get('https://www.wikidata.org/w/api.php', headers=headers, params=params)
-        if response.status_code == 200:
-            # pprint(response.headers)
-            data = response.json()
-            # pprint(data)
-            # Get the first QID
-            try:
-                qid = data["query"]["search"][0]["title"]
-                search_results = search_entities(
-                    qid,
-                    language="en",
-                    user_agent=config.user_agent,
-                    dict_result=True
-                )
-                if search_results is not None:
-                    first_result = search_results[0]
-                    # pprint(first_result)
-                    from models.suggestion import Suggestion
-                    item = Item(
-                        id=first_result["id"],
-                        label=first_result["label"],
-                        description=first_result["description"],
-                    )
-                    return Suggestion(
-                        item=item,
-                        ngram=self
-                    )
-            except IndexError:
-                logging.debug(f"No results found for {self.label}")
-        else:
-            print(f"Error got {response.status_code}: {response.text}")
-        # for result in search_results[0:1]:
-        #    pprint(result["description"])
-        # print("breaking")
-        # break
diff --git a/models/scholarly_articles.py b/models/scholarly_articles.py
deleted file mode 100644
index baaf74c..0000000
--- a/models/scholarly_articles.py
+++ /dev/null
@@ -1,71 +0,0 @@
-import logging
-
-from wikibaseintegrator.wbi_helpers import execute_sparql_query
-
-from helpers.console import console
-from models.suggestion import Suggestion
-from models.task import Task
-from models.wikidata import Labels, Items, Item
-
-
-class ScholarlyArticleLabels(Labels):
-    """This class has all code needed to fetch scientific articles, extract the labels,
-    clean them and find the top n-grams we need"""
-
-    def get_ngrams(self) -> dict:
-        # calculate_random_offset()
-        # console.print(f"Calculated offset is: {config.random_offset} and will be used to randomize "
-        #               f"the selection of scientific articles that are fetched")
-        self.fetch_labels_into_dataframe(quantity=5000,
-                                         # Fetch scientific articles without main subject
-                                         query="haswbstatement:P31=Q13442814 -haswbstatement:P921")
-        return self.extract_most_frequent_ngrams(quantity=7)
-
-
-class ScholarlyArticleItems(Items):
-    def fetch_based_on_label(self,
-                             suggestion: Suggestion = None,
-                             task: Task = None):
-        # logger = logging.getLogger(__name__)
-        if suggestion is None:
-            raise ValueError("suggestion was None")
-        if task is None:
-            raise ValueError("task was None")
-        # Fetch all items maching the search strings
-        self.list = []
-        for search_string in suggestion.search_strings:
-            results = execute_sparql_query(f'''
-            SELECT DISTINCT ?item ?itemLabel 
-            WHERE {{
-              hint:Query hint:optimizer "None".
-              SERVICE wikibase:mwapi {{
-                bd:serviceParam wikibase:api "Search";
-                                wikibase:endpoint "www.wikidata.org";
-                                # Include spaces around the n-gram to avoid edits like this one
-                                # https://www.wikidata.org/w/index.php?title=Q40671507&diff=1497186802&oldid=1496945583
-                                mwapi:srsearch 'haswbstatement:P31=Q13442814 -haswbstatement:P921={suggestion.item.id} "{search_string}"' .
-                ?title wikibase:apiOutput mwapi:title. 
-              }}
-              BIND(IRI(CONCAT(STR(wd:), ?title)) AS ?item)
-              ?item rdfs:label ?label.
-              # We lowercase and remove common symbols
-              BIND(REPLACE(LCASE(?label), ",", "") as ?label1)
-              BIND(REPLACE(?label1, ":", "") as ?label2)
-              BIND(REPLACE(?label2, ";", "") as ?label3)            
-              # We try matching beginning, middle and end
-              FILTER(CONTAINS(?label3, " {search_string.lower()} "@{task.language_code.value}) || 
-                     REGEX(?label3, ".* {search_string.lower()}$"@{task.language_code.value}) ||
-                     REGEX(?label3, "^{search_string.lower()} .*"@{task.language_code.value}))
-              # remove more specific forms of the main subject also
-              # Thanks to Jan Ainali for this improvement :)
-              MINUS {{?item wdt:P921 ?topic. ?topic wdt:P279 wd:{suggestion.item.id}. }}
-              SERVICE wikibase:label {{ bd:serviceParam wikibase:language "en". }}
-            }}
-            ''')
-            for item_json in results["results"]["bindings"]:
-                logging.debug(f"item_json:{item_json}")
-                item = Item(json=item_json)
-                self.list.append(item)
-            logging.info(f'Got {len(results["results"]["bindings"])} items from '
-                         f'WDQS using the search string {search_string}')
-        console.print(f"Got a total of {len(self.list)} items")
diff --git a/models/suggestion.py b/models/suggestion.py
deleted file mode 100644
index b7d0568..0000000
--- a/models/suggestion.py
+++ /dev/null
@@ -1,96 +0,0 @@
-import logging
-from typing import List
-from urllib.parse import quote
-
-from helpers.console import print_search_strings_table, console
-from helpers.enums import TaskIds
-from models.ngram import NGram
-from models.task import Task
-from models.wikidata import Item
-
-
-class Suggestion:
-    item: Item = None
-    ngram: NGram = None
-    search_strings: List[str] = None
-    task: Task = None
-    args = None
-
-    def __init__(self,
-                 item: Item = None,
-                 ngram: NGram = None,
-                 task: Task = None,
-                 args=None):
-        if item is None:
-            raise ValueError("item was None")
-        else:
-            self.item = item
-        if ngram is None:
-            raise ValueError("ngram was None")
-        else:
-            self.ngram: NGram = ngram
-        if task is None:
-            raise ValueError("task was None")
-        else:
-            self.task = task
-            self.args = args
-            self.extract_search_strings()
-
-    def __str__(self):
-        """Return label and description, the latter cut to 50 chars"""
-        if self.ngram is not None and self.item is not None:
-            return (
-                f"n-gram: [green][bold]{self.ngram.label}[/bold][/green]\n"
-                f"label: [bold]{self.item.label}[/bold]\n"
-                f"aliases: {', '.join(self.item.aliases)}\n"
-                f"description: {self.item.description[:70]}\n"
-                f"{self.item.url()}\n"
-                f"{self.ngram_search_url()}"
-            )
-        else:
-            if self.item is not None:
-                string = (
-                    f"label: [bold]{self.item.label}[/bold]\n"
-                    f"aliases: {', '.join(self.item.aliases)}\n"
-                    f"description: {self.item.description[:70]}\n"
-                    f"{self.item.url()}\n"
-                )
-                for url in self.search_urls():
-                    string = string + f"{url}\n"
-                return string
-
-    def search_urls(self) -> List[str]:
-        urls = []
-        for search_string in self.search_strings:
-            search_term = quote(f'"{search_string}"')
-            urls.append(f"https://www.wikidata.org/w/index.php?search={search_term}")
-        return urls
-
-    def ngram_search_url(self):
-        search_term = quote(f'"{self.ngram.label}"')
-        return f"https://www.wikidata.org/w/index.php?search={search_term}"
-
-    def extract_search_strings(self):
-        logger = logging.getLogger(__name__)
-        if self.args is None:
-            raise ValueError("args was None")
-        else:
-            logger.debug(f"args:{self.args}")
-            if self.args.no_aliases is True:
-                console.print("Alias matching is turned off")
-                no_aliases = True
-            else:
-                no_aliases = False
-        self.search_strings: List[str] = [self.item.label]
-        # Turn off alias matching for Riksdagen documents for now
-        if (
-            self.item.aliases is not None and
-            no_aliases is False and
-            self.task.id != TaskIds.RIKSDAGEN_DOCUMENTS
-        ):
-            for alias in self.item.aliases:
-                # logger.debug(f"extracting alias:{alias}")
-                self.search_strings.append(alias)
-        # logger.debug(f"search_strings:{self.search_strings}")
-        print_search_strings_table(self.search_strings)
-
diff --git a/models/task.py b/models/task.py
deleted file mode 100644
index 0c2effa..0000000
--- a/models/task.py
+++ /dev/null
@@ -1,38 +0,0 @@
-from helpers.enums import SupportedLanguageCode, TaskIds
-
-
-class Task:
-    """This class holds the tasks presented to the
-    user in the menu, the question asked when working
-    and the sparql query needed as a method"""
-    id: TaskIds
-    label: str
-    question: str
-    labels = None  # typing: Labels
-    language_code: SupportedLanguageCode
-
-    def __init__(self,
-                 id: TaskIds = None,
-                 label: str = None,
-                 question: str = None,
-                 labels=None,
-                 language_code: SupportedLanguageCode = None):
-        if id is None:
-            raise ValueError("Got no id")
-        if label is None:
-            raise ValueError("Got no label")
-        if question is None:
-            raise ValueError("Got no question")
-        if labels is None:
-            raise ValueError("Got no labels class")
-        if language_code is None:
-            raise ValueError("Got no language_code")
-        self.id = id
-        self.label = label
-        self.question = question
-        from models.wikidata import Labels
-        self.labels: Labels = labels
-        self.language_code = language_code
-
-    def __str__(self):
-        return f"{self.label}"
diff --git a/requirements.txt b/requirements.txt
index f006322..bef4108 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,7 +1,4 @@
 console-menu
-git+git://github.com/LeMyst/WikibaseIntegrator@v0.12.0.dev4#egg=wikibaseintegrator
-pandas~=1.3.2
-requests~=2.26.0
+git+git://github.com/LeMyst/WikibaseIntegrator@v0.12.0.dev5#egg=wikibaseintegrator
 rich~=10.9.0
-SPARQLWrapper~=1.8.5
-scikit-learn~=0.24.2
\ No newline at end of file
+SPARQLWrapper~=1.8.5
\ No newline at end of file
diff --git a/setup_environment.sh b/setup_environment.sh
new file mode 100644
index 0000000..a075c82
--- /dev/null
+++ b/setup_environment.sh
@@ -0,0 +1,3 @@
+# adjust this if necessary
+cd ~/itemsubjector
+pip install -r requirements.txt
diff --git a/src/__init__.py b/src/__init__.py
new file mode 100644
index 0000000..6d3d623
--- /dev/null
+++ b/src/__init__.py
@@ -0,0 +1,137 @@
+import argparse
+import logging
+from typing import List
+
+from wikibaseintegrator import wbi_login, wbi_config
+from wikibaseintegrator.wbi_helpers import execute_sparql_query
+
+import config
+from src.helpers.argparse_setup import setup_argparse_and_return_args
+from src.helpers.cleaning import strip_prefix
+from src.helpers.console import console, print_found_items_table, ask_add_to_job_queue, print_running_jobs, \
+    ask_yes_no_question, print_finished, \
+    print_keep_an_eye_on_wdqs_lag, print_best_practice, print_job_statistics, ask_discard_existing_job_pickle
+from src.helpers.enums import TaskIds
+from src.helpers.jobs import process_qid_into_job, process_user_supplied_qids_into_batch_jobs, run_jobs, \
+    handle_job_preparation_or_run_directly_if_any_jobs, get_validated_main_subjects_as_jobs
+from src.helpers.menus import select_task
+from src.helpers.migration import migrate_pickle_detection
+from src.helpers.pickle import parse_job_pickle, remove_job_pickle, add_to_job_pickle, check_if_pickle_exists, \
+    parse_main_subjects_pickle, get_hash_of_job_pickle
+from src.models.batch_job import BatchJob
+from src.models.riksdagen_documents import RiksdagenDocumentItems
+from src.models.scholarly_articles import ScholarlyArticleItems
+from src.models.suggestion import Suggestion
+from src.models.task import Task
+from src.models.thesis import ThesisItems
+from src.models.wikidata import Item
+from src.tasks import tasks
+
+logging.basicConfig(level=logging.WARNING)
+
+
+def login():
+    with console.status("Logging in with WikibaseIntegrator..."):
+        config.login_instance = wbi_login.Login(
+            auth_method='login',
+            user=config.username,
+            password=config.password,
+            debug=False
+        )
+        # Set User-Agent
+        wbi_config.config["USER_AGENT_DEFAULT"] = config.user_agent
+
+
+def match_existing_main_subjects(args: argparse.Namespace = None,
+                                 jobs: List[BatchJob] = None):
+    if jobs is None:
+        raise ValueError("jobs was None")
+    if not isinstance(jobs, List):
+        raise ValueError("jobs was not a list")
+    with console.status("Reading the main subjects file into memory"):
+        main_subjects = parse_main_subjects_pickle()
+    # raise Exception("debug exit")
+    jobs = get_validated_main_subjects_as_jobs(args=args,
+                                               main_subjects=main_subjects,
+                                               jobs=jobs)
+    handle_job_preparation_or_run_directly_if_any_jobs(args=args, jobs=jobs)
+
+
+def match_main_subjects_from_sparql(args: argparse.Namespace = None,
+                                    jobs: List[BatchJob] = None):
+    """Collect subjects via SPARQL and call get_validated_main_subjects()
+    If we get any validated jobs we handle them"""
+    if jobs is None:
+        raise ValueError("jobs was None")
+    if not isinstance(jobs, List):
+        raise ValueError("jobs was not a list")
+    with console.status("Running query on WDQS..."):
+        main_subjects = []
+        results = execute_sparql_query(args.sparql.replace("{", "{{").replace("}", "}}"),
+                                       debug=args.debug_sparql)
+        for item_json in results["results"]["bindings"]:
+            logging.debug(f"item_json:{item_json}")
+            main_subjects.append(item_json["item"]["value"])
+    if len(main_subjects) > 0:
+        console.print(f"Got {len(main_subjects)} results")
+        jobs = get_validated_main_subjects_as_jobs(
+            args=args,
+            main_subjects=main_subjects,
+            jobs=jobs
+        )
+        handle_job_preparation_or_run_directly_if_any_jobs(args=args, jobs=jobs)
+    else:
+        console.print("Got 0 results. Try another query or debug it using --debug")
+
+
+def main():
+    """This is the main function that makes everything else happen"""
+    logger = logging.getLogger(__name__)
+    migrate_pickle_detection()
+    jobs: List[BatchJob] = []
+    args = setup_argparse_and_return_args()
+    # console.print(args.list)
+    if args.remove_prepared_jobs is True:
+        remove_job_pickle()
+        console.print("Removed the job list.")
+        # exit(0)
+    if args.prepare_jobs is True:
+        logger.info("Preparing jobs")
+        if check_if_pickle_exists(config.job_pickle_file_path):
+            if not ask_discard_existing_job_pickle():
+                # the default is yes
+                # to avoid running batches multiple times by
+                # mistake (which does not harm Wikidata, but waste
+                # precious computing resources which we want to avoid.)
+                jobs = parse_job_pickle(silent=True)
+                if len(jobs) > 0:
+                    console.print(f"Found and loaded {len(jobs)} "
+                                  f"jobs with a total of "
+                                  f"{sum(len(job.items.list) for job in jobs)} items")
+            remove_job_pickle(silent=True)
+    if args.run_prepared_jobs is True:
+        logger.info("Running prepared jobs")
+        jobs = parse_job_pickle()
+        if jobs is not None and len(jobs) > 0:
+            file_hash = get_hash_of_job_pickle()
+            run_jobs(jobs)
+            # Remove the pickle afterwards
+            remove_job_pickle(hash=file_hash)
+    elif args.match_existing_main_subjects is True:
+        match_existing_main_subjects(args=args, jobs=jobs)
+    elif args.sparql:
+        match_main_subjects_from_sparql(args=args, jobs=jobs)
+    else:
+        # if not args.run_prepared_jobs:
+        if args.add is None:
+            console.print("Got no QIDs. Quitting")
+            exit(0)
+        task: Task = select_task()
+        if task is None:
+            raise ValueError("Got no task")
+        jobs.extend(process_user_supplied_qids_into_batch_jobs(args=args, task=task))
+        handle_job_preparation_or_run_directly_if_any_jobs(args=args, jobs=jobs)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/helpers/__init__.py b/src/helpers/__init__.py
similarity index 100%
rename from helpers/__init__.py
rename to src/helpers/__init__.py
diff --git a/src/helpers/argparse_setup.py b/src/helpers/argparse_setup.py
new file mode 100644
index 0000000..92afd5e
--- /dev/null
+++ b/src/helpers/argparse_setup.py
@@ -0,0 +1,94 @@
+import argparse
+
+
+def setup_argparse_and_return_args():
+    parser = argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter,
+                                     description="""
+ItemSubjector enables working main subject statements on items based on a 
+heuristic matching the subject with the title of the item. 
+
+Example adding one QID:
+'$ itemsubjector.py -l Q1234'
+
+Example adding one QID and prepare a job list to be run non-interactively later:
+'$ itemsubjector.py -l Q1234 -p'
+
+Example adding random QIDs from a list of main subjects extracted from 2 million scholarly articles:
+'$ itemsubjector.py -m'
+
+Example adding random QIDs from a list of main subjects extracted from 2 million scholarly articles 
+and prepare a job list:
+'$ itemsubjector.py -m -p'
+
+Example working on all diseases:
+'$ itemsubjector.py --sparql "SELECT ?item WHERE {?item wdt:P31 wd:Q12136.}"'
+    """)
+    parser.add_argument(
+        '-a', '--add', '--qid-to-add',
+        nargs='+',
+        help=('List of QIDs or URLs to Q-items that '
+              'are to be added as '
+              'main subjects on scientific articles. '
+              'Always add the most specific ones first. '
+              'See the README for examples')
+    )
+    parser.add_argument(
+        '-na', '--no-aliases',
+        action='store_true',
+        help='Turn off alias matching'
+    )
+    parser.add_argument(
+        '-nc', '--no-confirmation',
+        action='store_false',
+        default=True,
+        help='Turn off confirmation after displaying the search expressions, before running the queries.'
+    )
+    parser.add_argument(
+        '-p', '--prepare-jobs',
+        action='store_true',
+        help='Prepare a job for later execution, e.g. in a job engine'
+    )
+    parser.add_argument(
+        '-r', '--run-prepared-jobs',
+        action='store_true',
+        help='Run prepared jobs non-interactively'
+    )
+    parser.add_argument(
+        '-rm', '--remove-prepared-jobs',
+        action='store_true',
+        help='Remove prepared jobs'
+    )
+    parser.add_argument(
+        '-m', '--match-existing-main-subjects',
+        action='store_true',
+        help=('Match from list of 136.000 already used '
+              'main subjects on other scientific articles')
+    )
+    parser.add_argument(
+        '-w', '--limit-to-items-without-p921',
+        action='store_true',
+        help='Limit matching to scientific articles without P921 main subject'
+    )
+    parser.add_argument(
+        '-su', '--show-search-urls',
+        action='store_true',
+        help='Show an extra column in the table of search strings with links'
+    )
+    parser.add_argument(
+        '-iu', '--show-item-urls',
+        action='store_true',
+        help='Show an extra column in the table of items with links'
+    )
+    parser.add_argument(
+        '--sparql',
+        nargs='?',
+        help='Work on main subject items returned by this SPARQL query. '
+             'Note: "?item" has to be selected for it to work, see the example above.'
+    )
+    parser.add_argument(
+        '--debug-sparql',
+        action='store_true',
+        help='Enable debugging of SPARQL queries.',
+        default=False
+    )
+    return parser.parse_args()
diff --git a/helpers/calculations.py b/src/helpers/calculations.py
similarity index 100%
rename from helpers/calculations.py
rename to src/helpers/calculations.py
diff --git a/src/helpers/cleaning.py b/src/helpers/cleaning.py
new file mode 100644
index 0000000..0b1add0
--- /dev/null
+++ b/src/helpers/cleaning.py
@@ -0,0 +1,28 @@
+def strip_bad_chars(string):
+    # Note this has to match the cleaning done in the sparql query
+    # We lowercase and remove common symbols
+    # We replace like this to save CPU cycles see
+    # https://stackoverflow.com/questions/3411771/best-way-to-replace-multiple-characters-in-a-string
+    return (
+        string
+        # Needed for matching backslashes e.g. "Dmel\CG5330" on Q29717230
+        .replace("\\", "\\\\")
+        # Needed for when labels contain apostrophe
+        .replace("'", "\\'")
+        .replace(",", "")
+        .replace(":", "")
+        .replace(";", "")
+        .replace("(", "")
+        .replace(")", "")
+        .replace("[", "")
+        .replace("]", "")
+    )
+
+
+def strip_prefix(qid):
+    if "https://www.wikidata.org/wiki/" in qid:
+        qid = qid[30:]
+    if "http://www.wikidata.org/entity/" in qid:
+        qid = qid[31:]
+    # logger.debug(f"qid:{qid}")
+    return qid
\ No newline at end of file
diff --git a/src/helpers/console.py b/src/helpers/console.py
new file mode 100644
index 0000000..c54e6c7
--- /dev/null
+++ b/src/helpers/console.py
@@ -0,0 +1,131 @@
+import argparse
+from typing import List
+from urllib.parse import quote
+
+from rich.console import Console
+from rich.table import Table
+
+from src.models.batch_job import BatchJob
+from src.models.task import Task
+from src.models.wikidata import Items
+
+console = Console()
+
+
+def ask_yes_no_question(message: str):
+    # https://www.quora.com/
+    # I%E2%80%99m-new-to-Python-how-can-I-write-a-yes-no-question
+    # this will loop forever
+    while True:
+        answer = console.input(message + ' [Y/Enter/n]: ')
+        if len(answer) == 0 or answer[0].lower() in ('y', 'n'):
+            if len(answer) == 0:
+                return True
+            else:
+                # the == operator just returns a boolean,
+                return answer[0].lower() == 'y'
+
+
+def print_keep_an_eye_on_wdqs_lag():
+    console.print("Please keep an eye on the lag of the WDQS cluster here and avoid "
+                  "working if it is over a few minutes.\n"
+                  "https://grafana.wikimedia.org/d/000000489/wikidata-query-service?"
+                  "orgId=1&viewPanel=8&from=now-30m&to=now&refresh=1d "
+                  "You can see if any lagging servers are pooled here\n"
+                  "https://config-master.wikimedia.org/pybal/eqiad/wdqs\n"
+                  "If any enabled servers are lagging more than 5-10 minutes "
+                  "you can search phabricator for open tickets to see if the team is on it.\n"
+                  "If you don't find any feel free to create a new ticket like this:\n"
+                  "https://phabricator.wikimedia.org/T291621")
+
+
+def press_enter_to_continue():
+    console.input("Press Enter to continue.")
+
+
+def print_best_practice(task: Task):
+    if task.best_practice_information is not None:
+        console.print(task.best_practice_information)
+        press_enter_to_continue()
+
+
+def print_search_strings_table(args: argparse.Namespace = None,
+                               search_strings: List[str] = None):
+    if args is None:
+        raise ValueError("args was None")
+    if search_strings is None:
+        raise ValueError("search strings was None")
+    table = Table(title="Search strings")
+    table.add_column(f"Extracted the following {len(search_strings)} search strings")
+    if args.show_search_urls:
+        table.add_column(f"Wikidata search URL")
+    for string in search_strings:
+        if args.show_search_urls:
+            table.add_row(string, f"https://www.wikidata.org/w/index.php?search={quote(string)}")
+        else:
+            table.add_row(string)
+    console.print(table)
+
+
+def print_found_items_table(args: argparse.Namespace = None,
+                            items: Items = None):
+    if args is None:
+        raise ValueError("args was None")
+    if items is None:
+        raise ValueError("items was None")
+    table = Table(title="Matched items found")
+    if len(items.list) < 1000:
+        list_to_show = items.list[0:50]
+    else:
+        # Show 1 sample for each 20 items in the list
+        list_to_show = items.list[0:int(len(items.list) / 20)]
+    if len(items.list) > 4000:
+        console.print("[red]Warning: This is a very large batch, please proceed with caution[/red]")
+        press_enter_to_continue()
+    table.add_column(f"Showing a random subset of {len(list_to_show)} "
+                     f"items, please review as many as possible for false "
+                     f"positives and reject the batch if you find any.")
+    if args.show_item_urls:
+        table.add_column(f"Wikidata URL")
+    for item in list_to_show:
+        if args.show_item_urls:
+            # Fix rich parse bug with "[/TSUP]"
+            label = item.label.replace("[/", "['/")
+            table.add_row(label, item.url())
+        else:
+            table.add_row(item.label)
+    console.print(table)
+
+
+def ask_add_to_job_queue(job: BatchJob = None):
+    return ask_yes_no_question(f"Do you want to add this job for "
+                               f"[magenta]{job.suggestion.item.label}: "
+                               f"{job.suggestion.item.description}[/magenta] with "
+                               f"{len(job.items.list)} items to the queue? (see {job.suggestion.item.url()})")
+
+
+def print_running_jobs(jobs: List[BatchJob] = None):
+    if jobs is None:
+        raise ValueError("jobs was None")
+    console.print(f"Running {len(jobs)} job(s) with a total of "
+                  f"{sum(len(job.items.list) for job in jobs)} items "
+                  f"non-interactively now. You can take a "
+                  f"coffee break and lean back :)")
+
+
+def print_finished():
+    console.print("All jobs finished successfully")
+
+
+def print_job_statistics(jobs: List[BatchJob] = None):
+    if jobs is None:
+        raise ValueError("jobs was None")
+    console.print(f"The jobs list now contain a total of {len(jobs)} "
+                  f"jobs with a total of "
+                  f"{sum(len(job.items.list) for job in jobs)} items")
+
+
+def ask_discard_existing_job_pickle():
+    return ask_yes_no_question("A prepared list of jobs already exist, "
+                               "do you want to overwrite it? "
+                               "(pressing no will append to it)")
diff --git a/helpers/enums.py b/src/helpers/enums.py
similarity index 72%
rename from helpers/enums.py
rename to src/helpers/enums.py
index 6de34d4..0b9318e 100644
--- a/helpers/enums.py
+++ b/src/helpers/enums.py
@@ -2,10 +2,11 @@
 
 
 class SupportedLanguageCode(Enum):
-    ENGLISH="en"
-    SWEDISH="sv"
+    ENGLISH = "en"
+    SWEDISH = "sv"
 
 
 class TaskIds(Enum):
     SCHOLARLY_ARTICLES = auto()
     RIKSDAGEN_DOCUMENTS = auto()
+    THESIS = auto()
\ No newline at end of file
diff --git a/src/helpers/jobs.py b/src/helpers/jobs.py
new file mode 100644
index 0000000..763b928
--- /dev/null
+++ b/src/helpers/jobs.py
@@ -0,0 +1,170 @@
+from __future__ import annotations
+import argparse
+import random
+from datetime import datetime
+from typing import Union, List, TYPE_CHECKING
+
+from src import strip_prefix, print_best_practice, console, ask_yes_no_question, \
+    TaskIds, print_found_items_table, ask_add_to_job_queue, print_keep_an_eye_on_wdqs_lag, print_running_jobs, \
+    print_finished, print_job_statistics
+from src.tasks import tasks
+
+if TYPE_CHECKING:
+    from src import Task, BatchJob
+
+
+def process_qid_into_job(qid: str = None,
+                         task: Task = None,
+                         args: argparse.Namespace = None,
+                         confirmation: bool = False) -> Union[BatchJob, None]:
+    # logger = logging.getLogger(__name__)
+    if qid is None:
+        raise ValueError("qid was None")
+    if args is None:
+        raise ValueError("args was None")
+    if task is None:
+        raise ValueError("task was None")
+    from src import Item
+    item = Item(
+        id=strip_prefix(qid),
+        task=task
+    )
+    if item.label is not None:
+        console.print(f"Working on {item}")
+        # generate suggestion with all we need
+        from src import Suggestion
+        suggestion = Suggestion(
+            item=item,
+            task=task,
+            args=args
+        )
+        if confirmation:
+            answer = ask_yes_no_question("Do you want to continue?")
+            if not answer:
+                return None
+        with console.status(f'Fetching items with labels that have one of '
+                            f'the search strings by running a total of '
+                            f'{len(suggestion.search_strings) * task.number_of_queries_per_search_string} '
+                            f'queries on WDQS...'):
+            if task.id == TaskIds.SCHOLARLY_ARTICLES:
+                from src import ScholarlyArticleItems
+                items = ScholarlyArticleItems()
+            elif task.id == TaskIds.RIKSDAGEN_DOCUMENTS:
+                from src import RiksdagenDocumentItems
+                items = RiksdagenDocumentItems()
+            elif task.id == TaskIds.THESIS:
+                from src import ThesisItems
+                items = ThesisItems()
+            else:
+                raise ValueError(f"{task.id} was not recognized")
+            items.fetch_based_on_label(suggestion=suggestion,
+                                       task=task)
+        if len(items.list) > 0:
+            # Randomize the list
+            items.random_shuffle_list()
+            print_found_items_table(args=args,
+                                    items=items)
+            from src import BatchJob
+            job = BatchJob(
+                items=items,
+                suggestion=suggestion
+            )
+            answer = ask_add_to_job_queue(job)
+            if answer:
+                return job
+        else:
+            console.print("No matching items found")
+            return None
+    else:
+        console.print(f"Label for {task.language_code} was None on {item.url()}, skipping")
+
+
+def process_user_supplied_qids_into_batch_jobs(args: argparse.Namespace = None,
+                                               task: Task = None) -> List[BatchJob]:
+    """Given a list of QIDs, we go through
+    them and return a list of jobs"""
+    # logger = logging.getLogger(__name__)
+    if args is None:
+        raise ValueError("args was None")
+    if task is None:
+        raise ValueError("task was None")
+    print_best_practice(task)
+    jobs = []
+    for qid in args.add:
+        job = process_qid_into_job(qid=qid,
+                                   task=task,
+                                   args=args)
+        if job is not None:
+            jobs.append(job)
+    return jobs
+
+
+def run_jobs(jobs: List[BatchJob] = None):
+    if jobs is None:
+        raise ValueError("jobs was None")
+    print_keep_an_eye_on_wdqs_lag()
+    from src import login
+    login()
+    print_running_jobs(jobs)
+    count = 0
+    start_time = datetime.now()
+    for job in jobs:
+        count += 1
+        job.run(jobs=jobs, job_count=count)
+        console.print(f"runtime until now: {datetime.now() - start_time}")
+    print_finished()
+    end_time = datetime.now()
+    console.print(f'Total runtime: {end_time - start_time}')
+
+
+def handle_job_preparation_or_run_directly_if_any_jobs(args: argparse.Namespace = None,
+                                                       jobs: List[BatchJob] = None):
+    if len(jobs) > 0:
+        if args.prepare_jobs:
+            console.print(f"Adding {len(jobs)} job(s) to the jobs file")
+            for job in jobs:
+                from src import add_to_job_pickle
+                add_to_job_pickle(job)
+            print_job_statistics(jobs=jobs)
+            console.print(f"You can run the jobs "
+                          f"non-interactively e.g. on the Toolforge "
+                          f"Kubernetes cluster using -r or --run-prepared-jobs. "
+                          f"See Kubernetes_HOWTO.md for details.")
+        else:
+            run_jobs(jobs)
+
+
+def get_validated_main_subjects_as_jobs(args: argparse.Namespace = None,
+                                        main_subjects: List[str] = None,
+                                        jobs: List[BatchJob] = None):
+    """This function randomly picks a subject and present it for validation"""
+    # logger = logging.getLogger(__name__)
+    if jobs is None:
+        raise ValueError("jobs was None")
+    if not isinstance(jobs, List):
+        raise ValueError("jobs was not a list")
+    if args is None:
+        raise ValueError("args was None")
+    if main_subjects is None:
+        raise ValueError("main subjects was None")
+    # TODO implement better check for duplicates to avoid wasting resources
+    picked_before = []
+    while True:
+        console.print(f"Picking a random main subject")
+        qid = random.choice(main_subjects)
+        if qid not in picked_before:
+            job = process_qid_into_job(qid=qid,
+                                       # The scientific article task is hardcoded for now
+                                       task=tasks[0],
+                                       args=args,
+                                       confirmation=args.no_confirmation)
+            if job is not None:
+                jobs.append(job)
+                picked_before.append(qid)
+            print_job_statistics(jobs=jobs)
+            answer = ask_yes_no_question("Match one more?")
+            if not answer:
+                break
+        else:
+            console.print("Skipping already picked qid")
+    return jobs
diff --git a/helpers/menus.py b/src/helpers/menus.py
similarity index 90%
rename from helpers/menus.py
rename to src/helpers/menus.py
index dd39046..bd16101 100644
--- a/helpers/menus.py
+++ b/src/helpers/menus.py
@@ -3,8 +3,8 @@
 
 from consolemenu import SelectionMenu
 
-from models.suggestion import Suggestion
-from models.wikidata import WikimediaLanguageCode, Item
+from src.models.suggestion import Suggestion
+from src.models.wikidata import WikimediaLanguageCode, Item
 
 # def select_lexical_category():
 #     logger = logging.getLogger(__name__)
@@ -19,7 +19,7 @@
 #     logger.debug(f"selected:{selected_lexical_category_index}="
 #                  f"{selected_lexical_category}")
 #     return selected_lexical_category
-from tasks import tasks
+from src.tasks import tasks
 
 
 def select_language():
@@ -59,8 +59,8 @@ def select_suggestion(suggestions: List[Suggestion] = None,
     menu.join()
     selected_index = menu.selected_option
     selected_suggestion = None
-    if selected_index == len(suggestions) +1:
-        logger.debug("The user choosed to skip")
+    if selected_index == len(suggestions) + 1:
+        logger.debug("The user choose to skip")
     else:
         selected_suggestion = tasks[selected_index]
         logger.debug(f"selected:{selected_index}="
diff --git a/src/helpers/migration.py b/src/helpers/migration.py
new file mode 100644
index 0000000..084b6c4
--- /dev/null
+++ b/src/helpers/migration.py
@@ -0,0 +1,15 @@
+import config
+
+
+def migrate_pickle_detection():
+    try:
+        if config.job_pickle_file_path is None:
+            raise ValueError("the variable job_pickle_file_path in config "
+                             "has to contain a string like 'pickle.dat'")
+        if config.main_subjects_pickle_file_path is None:
+            raise ValueError("The variable main_subjects_pickle_file_path"
+                             "is None, see config.example.py")
+    except AttributeError:
+        raise ValueError("You need to migrate the new pickle variables"
+                         "in config.example.py to your config.py before "
+                         "you can continue")
diff --git a/src/helpers/pickle.py b/src/helpers/pickle.py
new file mode 100644
index 0000000..f2a5085
--- /dev/null
+++ b/src/helpers/pickle.py
@@ -0,0 +1,102 @@
+import os
+import hashlib
+import pickle
+from typing import List
+
+import config
+from src.helpers.console import console
+from src.models.batch_job import BatchJob
+
+
+def add_to_job_pickle(job: BatchJob = None):
+    if job is None:
+        raise ValueError("Job was None")
+    else:
+        with open(config.job_pickle_file_path, 'ab') as file:
+            pickle.dump(job, file, pickle.DEFAULT_PROTOCOL)
+
+
+def add_to_main_subject_pickle(subjects: List[str] = None):
+    with open(config.main_subjects_pickle_file_path, 'wb') as file:
+        for qid in subjects:
+            pickle.dump(qid, file, pickle.DEFAULT_PROTOCOL)
+
+
+def read_from_pickle(path):
+    with open(path, 'rb') as file:
+        try:
+            while True:
+                yield pickle.load(file)
+        except EOFError:
+            pass
+
+
+def check_if_pickle_exists(path):
+    if os.path.exists(path):
+        return True
+    else:
+        return False
+
+
+def parse_job_pickle(silent: bool = False) -> List[BatchJob]:
+    """Reads the pickle into a list of batch jobs"""
+    if check_if_pickle_exists(config.job_pickle_file_path):
+        jobs: List[BatchJob] = []
+        for job in read_from_pickle(config.job_pickle_file_path):
+            jobs.append(job)
+        if len(jobs) == 0:
+            if not silent:
+                console.print("No prepared jobs found")
+        else:
+            return jobs
+    else:
+        if not silent:
+            console.print("No pickle file found")
+
+
+def parse_main_subjects_pickle() -> List[str]:
+    """Reads the pickle into a list of main subjects"""
+    if check_if_pickle_exists(config.main_subjects_pickle_file_path):
+        subjects = []
+        for subject in read_from_pickle(config.main_subjects_pickle_file_path):
+            subjects.append(subject)
+        if len(subjects) == 0:
+            console.print("No qids found in the pickle.")
+        else:
+            # print(f"found:{subjects}")
+            return subjects
+    else:
+        console.print("No main subjects pickle file found. "
+                      "Create it by running 'python fetch_main_subjects.py'")
+        exit(0)
+
+
+def remove_job_pickle(silent: bool = False,
+                      hash: str = None):
+    if hash is None:
+        if os.path.exists(config.job_pickle_file_path):
+            os.remove(config.job_pickle_file_path)
+            if not silent:
+                console.print("The job list file was removed")
+    if os.path.exists(config.job_pickle_file_path):
+        hash_now = get_hash_of_job_pickle()
+        if hash == hash_now:
+            os.remove(config.job_pickle_file_path)
+            if not silent:
+                console.print("The job list file was removed")
+        else:
+            console.print("Job list file not deleted because the contents "
+                          "has changed since this batch of jobs was started.")
+    else:
+        console.print(f"Could not delete the job file. No file found at {config.job_pickle_file_path}")
+
+
+def get_hash_of_job_pickle():
+    # inspired by https://codezup.com/python-program-calculate-hash-of-file-hashlib/
+    block_size = 65536  # lets read stuff in 64kb chunks!
+    hasher = hashlib.md5()
+    with open(config.job_pickle_file_path, 'rb') as file:
+        buf = file.read(block_size)
+        hasher.update(buf)
+    # print(hasher.hexdigest())
+    return hasher.hexdigest()
diff --git a/models/__init__.py b/src/models/__init__.py
similarity index 100%
rename from models/__init__.py
rename to src/models/__init__.py
diff --git a/src/models/batch_job.py b/src/models/batch_job.py
new file mode 100644
index 0000000..b96c5b4
--- /dev/null
+++ b/src/models/batch_job.py
@@ -0,0 +1,21 @@
+from __future__ import annotations
+from dataclasses import dataclass
+from typing import List, TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from src.models.suggestion import Suggestion
+    from src.models.wikidata import Items
+
+
+@dataclass
+class BatchJob:
+    """Models a batch job intended to be run non-interactively"""
+    suggestion: Suggestion
+    items: Items
+
+    def run(self, jobs: List[BatchJob], job_count: int = None):
+        if jobs is None:
+            raise ValueError("jobs was None")
+        if job_count is None:
+            raise ValueError("job count was None")
+        self.suggestion.add_to_items(items=self.items, jobs=jobs, job_count=job_count)
diff --git a/models/riksdagen_documents.py b/src/models/riksdagen_documents.py
similarity index 66%
rename from models/riksdagen_documents.py
rename to src/models/riksdagen_documents.py
index d226a63..f6ff640 100644
--- a/models/riksdagen_documents.py
+++ b/src/models/riksdagen_documents.py
@@ -2,24 +2,10 @@
 
 from wikibaseintegrator.wbi_helpers import execute_sparql_query
 
-from helpers.console import console
-from models.suggestion import Suggestion
-from models.task import Task
-from models.wikidata import Labels, Items, Item
-
-
-class RiksdagenDocumentLabels(Labels):
-    """This class has all code needed to fetch Riksdagen documents, extract the labels,
-    clean them and find the top n-grams we need"""
-
-    # def get_ngrams(self) -> dict:
-    #     # calculate_random_offset()
-    #     # console.print(f"Calculated offset is: {config.random_offset} and will be used to randomize "
-    #     #               f"the selection of scientific articles that are fetched")
-    #     self.fetch_labels_into_dataframe(quantity=5000,
-    #                                      # Fetch Riksdagen documents without main subject
-    #                                      query="haswbstatement:P8433 -haswbstatement:P921")
-    #     return self.extract_most_frequent_ngrams(quantity=7)
+from src.helpers.console import console
+from src.models.suggestion import Suggestion
+from src.models.task import Task
+from src.models.wikidata import Items, Item
 
 
 class RiksdagenDocumentItems(Items):
@@ -29,10 +15,16 @@ def fetch_based_on_label(self,
         # logger = logging.getLogger(__name__)
         if suggestion is None:
             raise ValueError("suggestion was None")
+        if suggestion.args.limit_to_items_without_p921:
+            raise Exception("Limiting to items without P921 is not "
+                            "supported yet for this task.")
         if task is None:
             raise ValueError("task was None")
         # Fetch all items maching the search strings
         self.list = []
+        # Include spaces around the n-gram to avoid edits like this one
+        # https://www.wikidata.org/w/index.php?title=Q40671507&diff=1497186802&oldid=1496945583
+        # Lowercase is not needed here as Elastic matches anyway
         for search_string in suggestion.search_strings:
             results = execute_sparql_query(f'''
             SELECT DISTINCT ?item ?itemLabel 
@@ -41,9 +33,6 @@ def fetch_based_on_label(self,
               SERVICE wikibase:mwapi {{
                 bd:serviceParam wikibase:api "Search";
                                 wikibase:endpoint "www.wikidata.org";
-                                # Include spaces around the n-gram to avoid edits like this one
-                                # https://www.wikidata.org/w/index.php?title=Q40671507&diff=1497186802&oldid=1496945583
-                                # Lowercase is not needed here as Elastic matches anyway
                                 mwapi:srsearch 'haswbstatement:P8433 -haswbstatement:P921={suggestion.item.id} "{search_string}"' .
                 ?title wikibase:apiOutput mwapi:title. 
               }}
@@ -59,7 +48,7 @@ def fetch_based_on_label(self,
               MINUS {{?item wdt:P921 ?topic. ?topic wdt:P279 wd:{suggestion.item.id}. }}
               SERVICE wikibase:label {{ bd:serviceParam wikibase:language "sv". }}
             }}
-            ''')
+            ''', debug=suggestion.args.debug_sparql)
             for item_json in results["results"]["bindings"]:
                 logging.debug(f"item_json:{item_json}")
                 item = Item(json=item_json,
diff --git a/src/models/scholarly_articles.py b/src/models/scholarly_articles.py
new file mode 100644
index 0000000..7853257
--- /dev/null
+++ b/src/models/scholarly_articles.py
@@ -0,0 +1,126 @@
+import logging
+
+from wikibaseintegrator.wbi_helpers import execute_sparql_query
+
+from src.helpers.cleaning import strip_bad_chars
+from src.helpers.console import console
+from src.models.suggestion import Suggestion
+from src.models.task import Task
+from src.models.wikidata import Items, Item
+
+
+def build_query(suggestion: Suggestion = None,
+                search_string: str = None,
+                task: Task = None,
+                cirrussearch_parameters: str = None):
+    if suggestion is None:
+        raise ValueError("suggestion was None")
+    if search_string is None:
+        raise ValueError("search_string was None")
+    if task is None:
+        raise ValueError("task was None")
+    if cirrussearch_parameters is None:
+        raise ValueError("cirrussearch_parameters was None")
+    # This query uses https://www.w3.org/TR/sparql11-property-paths/ to
+    # find subjects that are subclass of one another up to 3 hops away
+    # This query also uses the https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual/MWAPI
+    # which has a hardcoded limit of 10,000 items so you will never get more matches than that
+    # This query use regex to match beginning, middle and end of the label of matched items
+    # The replacing lines should match the similar python replacements in cleaning.py
+    # The replacing with "\\\\\\\\" becomes "\\\\" after leaving python and then it works in
+    # SPARQL where it becomes "\\" and thus match a single backslash
+    return (f"""
+        SELECT DISTINCT ?item ?itemLabel 
+        WHERE {{
+          hint:Query hint:optimizer "None".
+          BIND(STR('{cirrussearch_parameters} \"{search_string}\"') as ?search_string)
+          SERVICE wikibase:mwapi {{
+            bd:serviceParam wikibase:api "Search";
+                            wikibase:endpoint "www.wikidata.org";
+                            mwapi:srsearch ?search_string.
+            ?title wikibase:apiOutput mwapi:title. 
+          }}
+          BIND(IRI(CONCAT(STR(wd:), ?title)) AS ?item)
+          ?item rdfs:label ?label.
+          BIND(REPLACE(LCASE(?label), ",", "") as ?label1)
+          BIND(REPLACE(?label1, ":", "") as ?label2)
+          BIND(REPLACE(?label2, ";", "") as ?label3)          
+          BIND(REPLACE(?label3, "\\\\(", "") as ?label4)
+          BIND(REPLACE(?label4, "\\\\)", "") as ?label5)
+          BIND(REPLACE(?label5, "\\\\[", "") as ?label6)
+          BIND(REPLACE(?label6, "\\\\]", "") as ?label7)
+          BIND(REPLACE(?label7, "\\\\\\\\", "") as ?label8)
+          BIND(?label8 as ?cleaned_label)
+          FILTER(CONTAINS(?cleaned_label, ' {search_string.lower()} '@{task.language_code.value}) || 
+                 REGEX(?cleaned_label, '.* {search_string.lower()}$'@{task.language_code.value}) ||
+                 REGEX(?cleaned_label, '^{search_string.lower()} .*'@{task.language_code.value}))
+          MINUS {{?item wdt:P921/wdt:P279 wd:{suggestion.item.id}. }}
+          MINUS {{?item wdt:P921/wdt:P279/wdt:P279 wd:{suggestion.item.id}. }}
+          MINUS {{?item wdt:P921/wdt:P279/wdt:P279/wdt:P279 wd:{suggestion.item.id}. }}
+          SERVICE wikibase:label {{ bd:serviceParam wikibase:language "en". }}
+        }}
+        """)
+
+
+def process_results(results):
+    items = []
+    for item_json in results["results"]["bindings"]:
+        logging.debug(f"item_json:{item_json}")
+        item = Item(json=item_json)
+        items.append(item)
+    return items
+
+
+class ScholarlyArticleItems(Items):
+    """This supports both published peer reviewed articles and preprints"""
+    def fetch_based_on_label(self,
+                             suggestion: Suggestion = None,
+                             task: Task = None):
+        # logger = logging.getLogger(__name__)
+        if suggestion is None:
+            raise ValueError("suggestion was None")
+        if task is None:
+            raise ValueError("task was None")
+        if suggestion.args.limit_to_items_without_p921:
+            console.print("Limiting to scholarly articles without P921 main subject only")
+            cirrussearch_parameters = f"haswbstatement:P31=Q13442814 -haswbstatement:P921"
+        else:
+            cirrussearch_parameters = f"haswbstatement:P31=Q13442814 -haswbstatement:P921={suggestion.item.id}"
+        # Fetch all items matching the search strings
+        self.list = []
+        for search_string in suggestion.search_strings:
+            search_string = strip_bad_chars(search_string)
+            results = execute_sparql_query(
+                build_query(
+                    cirrussearch_parameters=cirrussearch_parameters,
+                    suggestion=suggestion,
+                    search_string=search_string,
+                    task=task)
+            )
+            logging.info(f'Got {len(results["results"]["bindings"])} scholarly items from '
+                         f'WDQS using the search string {search_string}')
+            self.list.extend(process_results(results))
+            # preprints
+            # We don't use CirrusSearch in this query because we can do it more easily in
+            # SPARQL on a small subgraph like this
+            # find all items that are ?item wdt:P31/wd:P279* wd:Q1266946
+            # minus the QID we want to add
+            results_preprint = execute_sparql_query(f'''
+                SELECT DISTINCT ?item ?itemLabel 
+                WHERE {{
+                  ?item wdt:P31/wd:P279* wd:Q580922. # preprint 
+                  MINUS {{
+                  ?item wdt:P921 wd:{suggestion.item.id};
+                  }}
+                  ?item rdfs:label ?label.
+                  FILTER(CONTAINS(LCASE(?label), " {search_string.lower()} "@{task.language_code.value}) || 
+                         REGEX(LCASE(?label), ".* {search_string.lower()}$"@{task.language_code.value}) ||
+                         REGEX(LCASE(?label), "^{search_string.lower()} .*"@{task.language_code.value}))
+                  MINUS {{?item wdt:P921/wdt:P279 wd:{suggestion.item.id}. }}
+                  SERVICE wikibase:label {{ bd:serviceParam wikibase:language "en". }}
+                }}
+                ''', debug=suggestion.args.debug_sparql)
+            logging.info(f'Got {len(results["results"]["bindings"])} preprint items from '
+                         f'WDQS using the search string {search_string}')
+            self.list.extend(process_results(results_preprint))
+        console.print(f"Got a total of {len(self.list)} items")
diff --git a/src/models/suggestion.py b/src/models/suggestion.py
new file mode 100644
index 0000000..a26cd85
--- /dev/null
+++ b/src/models/suggestion.py
@@ -0,0 +1,127 @@
+import argparse
+import logging
+from typing import List
+from urllib.parse import quote
+
+from wikibaseintegrator.datatypes import Item as ItemType
+
+from src.helpers.calculations import calculate_random_editgroups_hash
+from src.helpers.console import print_search_strings_table, console
+from src.helpers.enums import TaskIds
+from src.models.batch_job import BatchJob
+from src.models.task import Task
+from src.models.wikidata import Item, Items
+
+
+class Suggestion:
+    item: Item = None
+    search_strings: List[str] = None
+    task: Task = None
+    args: argparse.Namespace = None
+
+    def __init__(self,
+                 item: Item = None,
+                 task: Task = None,
+                 args=None):
+        if item is None:
+            raise ValueError("item was None")
+        else:
+            self.item = item
+        if task is None:
+            raise ValueError("task was None")
+        else:
+            self.task = task
+            self.args = args
+            self.extract_search_strings()
+
+    def __str__(self):
+        """Return label and description, the latter cut to 50 chars"""
+        if self.item is not None:
+            string = (
+                f"label: [bold]{self.item.label}[/bold]\n"
+                f"aliases: {', '.join(self.item.aliases)}\n"
+                f"description: {self.item.description[:70]}\n"
+                f"{self.item.url()}\n"
+            )
+            for url in self.search_urls():
+                string = string + f"{url}\n"
+            return string
+
+    def add_to_items(self,
+                     items: Items = None,
+                     jobs: List[BatchJob] = None,
+                     job_count: int = None):
+        """Add a suggested QID as main subject on all items that
+        have a label that matches one of the search strings for this QID
+        We calculate a new edit group hash each time this function is
+        called so similar edits are grouped and easily be undone.
+
+        This function is non-interactive"""
+        if items is None:
+            raise ValueError("Items was None")
+        if jobs is None:
+            raise ValueError("jobs was None")
+        if job_count is None:
+            raise ValueError("job count was None")
+        editgroups_hash: str = calculate_random_editgroups_hash()
+        count = 0
+        for target_item in items.list:
+            count += 1
+            with console.status(f"Uploading main subject [green]{self.item.label}[/green] to {target_item.label}"):
+                main_subject_property = "P921"
+                reference = ItemType(
+                    "Q69652283",  # inferred from title
+                    prop_nr="P887"  # based on heuristic
+                )
+                statement = ItemType(
+                    self.item.id,
+                    prop_nr=main_subject_property,
+                    references=[reference]
+                )
+                target_item.upload_one_statement_to_wikidata(
+                    statement=statement,
+                    summary=f"[[Property:{main_subject_property}]]: [[{self.item.id}]]",
+                    editgroups_hash=editgroups_hash
+                )
+            console.print(f"(job {job_count}/{len(jobs)})(item {count}/{len(items.list)}) "
+                          f"Added '{self.item.label}' to {target_item.label}: {target_item.url()}")
+            # input("Press enter to continue")
+
+    def extract_search_strings(self):
+        def clean_special_symbols(string: str):
+            return string.replace("®", "").replace("™", "")
+
+        logger = logging.getLogger(__name__)
+        if self.args is None:
+            raise ValueError("args was None")
+        else:
+            logger.debug(f"args:{self.args}")
+            if self.args.no_aliases is True:
+                console.print("Alias matching is turned off")
+                no_aliases = True
+            else:
+                no_aliases = False
+        self.search_strings: List[str] = [clean_special_symbols(self.item.label)]
+        if (
+            self.item.aliases is not None and
+            no_aliases is False
+        ):
+            for alias in self.item.aliases:
+                # logger.debug(f"extracting alias:{alias}")
+                if len(alias) < 5:
+                    console.print(f"Skipping short alias '{alias}' to avoid false positives", style="#FF8000")
+                else:
+                    self.search_strings.append(clean_special_symbols(alias))
+        # logger.debug(f"search_strings:{self.search_strings}")
+        print_search_strings_table(args=self.args,
+                                   search_strings=self.search_strings)
+
+    def search_urls(self) -> List[str]:
+        urls = []
+        for search_string in self.search_strings:
+            search_term = quote(f'"{search_string}"')
+            urls.append(f"https://www.wikidata.org/w/index.php?search={search_term}")
+        return urls
+
+
+
diff --git a/src/models/task.py b/src/models/task.py
new file mode 100644
index 0000000..1260025
--- /dev/null
+++ b/src/models/task.py
@@ -0,0 +1,37 @@
+from typing import Union
+
+from src.helpers.enums import SupportedLanguageCode, TaskIds
+
+
+# console-menu does not support dataclass (yet)
+# @dataclass
+class Task:
+    """This class holds the tasks presented to the
+    user in the menu and related data"""
+    best_practice_information: Union[str, None] = None
+    id: TaskIds = None
+    label: str = None
+    language_code: SupportedLanguageCode = None
+    number_of_queries_per_search_string = 1
+
+    def __init__(self,
+                 best_practice_information: str = None,
+                 id: TaskIds = None,
+                 label: str = None,
+                 language_code: SupportedLanguageCode = None,
+                 number_of_queries_per_search_string: int = None):
+        if id is None:
+            raise ValueError("Got no id")
+        if label is None:
+            raise ValueError("Got no label")
+        if language_code is None:
+            raise ValueError("Got no language_code")
+        self.id = id
+        self.label = label
+        self.language_code = language_code
+        self.best_practice_information = best_practice_information
+        if number_of_queries_per_search_string is not None:
+            self.number_of_queries_per_search_string = number_of_queries_per_search_string
+
+    def __str__(self):
+        return f"{self.label}"
diff --git a/src/models/thesis.py b/src/models/thesis.py
new file mode 100644
index 0000000..5569ae9
--- /dev/null
+++ b/src/models/thesis.py
@@ -0,0 +1,62 @@
+import logging
+
+from wikibaseintegrator.wbi_helpers import execute_sparql_query
+
+from src.helpers.console import console
+from src.models.suggestion import Suggestion
+from src.models.task import Task
+from src.models.wikidata import Items, Item
+
+# There were ~16.000 thesis' in WD when this was written
+
+
+class ThesisItems(Items):
+    def fetch_based_on_label(self,
+                             suggestion: Suggestion = None,
+                             task: Task = None):
+        # logger = logging.getLogger(__name__)
+        if suggestion is None:
+            raise ValueError("suggestion was None")
+        if suggestion.args.limit_to_items_without_p921:
+            raise Exception("Limiting to items without P921 is not "
+                            "supported yet for this task.")
+        if task is None:
+            raise ValueError("task was None")
+        # Fetch all items maching the search strings
+        self.list = []
+        for search_string in suggestion.search_strings:
+            # We don't use CirrusSearch in this query because we can do it more easily in
+            # SPARQL on a small subgraph like this
+            # find all items that are ?item wdt:P31/wd:P279* wd:Q1266946
+            # minus the QID we want to add
+            results = execute_sparql_query(f'''
+            SELECT DISTINCT ?item ?itemLabel 
+            WHERE {{
+              {{
+                ?item wdt:P31/wd:P279* wd:Q1266946. # thesis
+              }} UNION
+              {{
+                ?item wdt:P31/wd:P279* wd:Q1385450. # dissertation
+              }} UNION
+              {{
+                ?item wdt:P31/wd:P279* wd:Q3099732. # technical report
+              }}
+              MINUS {{
+              ?item wdt:P921 wd:{suggestion.item.id};
+              }}
+              ?item rdfs:label ?label.
+              FILTER(CONTAINS(LCASE(?label), " {search_string.lower()} "@{task.language_code.value}) || 
+                     REGEX(LCASE(?label), ".* {search_string.lower()}$"@{task.language_code.value}) ||
+                     REGEX(LCASE(?label), "^{search_string.lower()} .*"@{task.language_code.value}))
+              MINUS {{?item wdt:P921 ?topic. ?topic wdt:P279 wd:{suggestion.item.id}. }}
+              SERVICE wikibase:label {{ bd:serviceParam wikibase:language "en". }}
+            }}
+            ''', debug=suggestion.args.debug_sparql)
+            for item_json in results["results"]["bindings"]:
+                logging.debug(f"item_json:{item_json}")
+                item = Item(json=item_json,
+                            task=task)
+                self.list.append(item)
+            logging.info(f'Got {len(results["results"]["bindings"])} items from '
+                         f'WDQS using the search string {search_string}')
+        console.print(f"Got a total of {len(self.list)} items")
diff --git a/models/wikidata.py b/src/models/wikidata.py
similarity index 84%
rename from models/wikidata.py
rename to src/models/wikidata.py
index 224b1df..7e2aeac 100644
--- a/models/wikidata.py
+++ b/src/models/wikidata.py
@@ -3,23 +3,17 @@
 """
 import logging
 import random
-import re
 from enum import Enum
-from typing import List, Dict
+from typing import List
 
-import pandas as pd
-from pandas import DataFrame
-from sklearn.feature_extraction.text import CountVectorizer
 from wikibaseintegrator import wbi_config, WikibaseIntegrator
 from wikibaseintegrator.datatypes import BaseDataType
 from wikibaseintegrator.models import Alias
 from wikibaseintegrator.wbi_enums import ActionIfExists
 
 import config
-from helpers.sparql_dataframe import query_wikidata
-
 # We get the URL for the Wikibase from here
-from models.task import Task
+from src.models.task import Task
 
 wbi_config.config['USER_AGENT'] = config.user_agent
 
@@ -793,9 +787,9 @@ class Item(Entity):
     """This represents an item in Wikidata
     We always work on one language at a time,
     so don't bother with languages here and keep to simple strings"""
-    id: str
-    label: str
-    description: str
+    id: str =  None
+    label: str = None
+    description: str = None
     aliases: List[str] = None
 
     def __init__(self,
@@ -810,9 +804,9 @@ def __init__(self,
         else:
             if id is not None:
                 self.id = str(EntityID(id))
-            if label is None and aliases is None:
+            if description is None and label is None and aliases is None:
                 logging.debug("here now")
-                self.fetch_label_and_aliases(task=task)
+                self.fetch_label_and_description_and_aliases(task=task)
             elif label is None or aliases is None:
                 raise ValueError("This is not supported. "
                                  "Either both state the label and "
@@ -820,24 +814,24 @@ def __init__(self,
             else:
                 self.label = label
                 self.aliases = aliases
-            self.description = description
+                self.description = description
 
     def __str__(self):
-        return f"{self.id}: {self.label}"
+        return f"{self.label}, see {self.url()}"
 
     def parse_json(self, json):
         """Parse the WDQS json"""
         logger = logging.getLogger(__name__)
         try:
-            logger.info(json["item"])
+            logger.debug(f'item_json:{json["item"]}')
             self.id = str(EntityID(json["item"]["value"]))
         except KeyError:
             pass
         try:
-            logger.info(json["itemLabel"])
+            logger.debug(json["itemLabel"])
             self.label = (json["itemLabel"]["value"])
         except KeyError:
-            pass
+            logger.info(f"no label found")
 
     def parse_from_wdqs_json(self, json):
         """Parse the json into the object"""
@@ -848,16 +842,21 @@ def parse_from_wdqs_json(self, json):
             if variable == "itemLabel":
                 self.label = variable
 
-    def fetch_label_and_aliases(self,
-                                task: Task = None):
+    def fetch_label_and_description_and_aliases(self,
+                                                task: Task = None):
         """Fetch label and aliases in the task language from the Wikidata API"""
         if task is None:
             raise ValueError("task was None")
-        from helpers.console import console
+        from src.helpers.console import console
         with console.status(f"Fetching {task.language_code.value} label and aliases..."):
             wbi = WikibaseIntegrator()
             item = wbi.item.get(self.id)
-            self.label = str(item.labels.get(task.language_code.value))
+            label = item.labels.get(task.language_code.value)
+            if label is not None:
+                self.label = str(label)
+            description = item.descriptions.get(task.language_code.value)
+            if description is not None:
+                self.description = str(description)
             aliases: List[Alias] = item.aliases.get(task.language_code.value)
             # logging.debug(f"aliases from wbi:{item.aliases.get('en')}")
             if aliases is not None:
@@ -868,114 +867,6 @@ def fetch_label_and_aliases(self,
                 # logging.debug(f"aliases:{self.aliases}")
 
 
-class Labels:
-    dataframe: DataFrame = None
-    document_term_matrix: DataFrame = None
-
-    def fetch_labels_into_dataframe(self,
-                                    quantity: int = None,
-                                    query: str = None):
-        logger = logging.getLogger(__name__)
-        if query is None:
-            raise ValueError("Get no query")
-        if quantity is None:
-            raise ValueError("Get no quantity")
-        if config.random_offset is None:
-            raise ValueError("No random offset present in the config")
-        from helpers.console import console
-        with console.status(f"Fetching {quantity} labels..."):
-            dataframe = (query_wikidata(f'''
-                #author:So9q inspired a query by Azertus
-                #date:2021-09-11
-                SELECT #?item 
-                ?itemLabel 
-                WHERE {{
-                    {{ SELECT * WHERE {{
-                      SERVICE wikibase:mwapi {{
-                        bd:serviceParam wikibase:endpoint "www.wikidata.org";
-                                        wikibase:api "Search";
-                                        # scientific article without main subject
-                                        mwapi:srsearch '{query}'; 
-                                        mwapi:language "en".
-                        ?title wikibase:apiOutput mwapi:title. 
-                      }}
-                      BIND(URI(CONCAT('http://www.wikidata.org/entity/', ?title)) AS ?item)
-                    }} 
-                    OFFSET {config.random_offset}
-                    LIMIT {quantity}
-                    }}  
-                  SERVICE wikibase:label {{
-                    bd:serviceParam wikibase:language "en" .
-                    ?item rdfs:label ?itemLabel .
-                  }}
-                }}
-            ''', config.endpoint))
-            quantity = len(dataframe)
-            if quantity > 0:
-                console.print(f"Got {quantity} items with English labels")
-                # remove unwanted columns
-                dataframe = dataframe[["itemLabel.value"]]
-                # rename column
-                dataframe.rename(columns={'itemLabel.value': 'label'}, inplace=True)
-                # debug
-                logger.debug(dataframe.head())
-                self.dataframe = dataframe
-            else:
-                raise ValueError("Got no data from WDQS")
-
-    def clean_labels(self):
-        if self.dataframe is None:
-            raise Exception("No dataframe found")
-        # forked from: https://github.com/adashofdata/nlp-in-python-tutorial/blob/master/1-Data-Cleaning.ipynb
-        # Apply a first round of text cleaning techniques
-
-        def clean_text(text):
-            """Make text lowercase, remove text in square brackets,
-            remove punctuation and remove words containing numbers."""
-            # text = text.lower()
-            text = re.sub('\[.*?\]', '', text)
-            text = re.sub(',', '', text)  # remove commas
-            text = re.sub(':', '', text)  # remove colons
-            text = re.sub('\.', '', text)  # remove full stop
-            text = re.sub('\?', '', text)  # remove question mark
-            text = re.sub(' - ', '', text)  # remove lone dash
-            text = re.sub('\w*\d\w*', '', text)
-            return text
-        self.clean_section = pd.DataFrame(
-            self.dataframe.label.apply(lambda x: clean_text(x))
-        )
-
-    def create_document_term_matrix(self):
-        logger = logging.getLogger(__name__)
-        self.clean_labels()
-        # We are going to create a document-term matrix using CountVectorizer, and exclude common English stop words
-        cv = CountVectorizer(stop_words='english',
-                             min_df=3,  # minimum frequency threshold
-                             ngram_range=(2, 3))
-        data_cv = cv.fit_transform(self.dataframe.label)
-        self.document_term_matrix = pd.DataFrame(data_cv.toarray(), columns=cv.get_feature_names())
-        logger.debug(self.document_term_matrix)
-
-    def extract_most_frequent_ngrams(self, quantity: int = None) -> Dict:
-        """This extracts the most frequent n-grams from our Document Term Matrix"""
-        logger = logging.getLogger(__name__)
-        if quantity is None:
-            raise ValueError("quantity was None")
-        self.create_document_term_matrix()
-        top_dict = {}
-        # We want the dict to be structured like this:
-        # key: n-gram
-        # value: frequency
-        for column in self.document_term_matrix.columns:
-            top_dict[column] = self.document_term_matrix[column].sum()
-            # break
-        logger.info(f"top_dict size:{len(top_dict)}")
-        # Sort descending
-        sorted_list = sorted(top_dict.items(), key=lambda x: x[1], reverse=True)
-        # sorted_dict
-        return dict(sorted_list[0:quantity])
-
-
 class Items:
     list: List[Item] = []
 
diff --git a/src/tasks.py b/src/tasks.py
new file mode 100644
index 0000000..6621871
--- /dev/null
+++ b/src/tasks.py
@@ -0,0 +1,51 @@
+from src.helpers.enums import SupportedLanguageCode, TaskIds
+from src.models.task import Task
+
+# When adding a new task, also add it in the enum
+
+tasks = [
+    Task(
+        id=TaskIds.SCHOLARLY_ARTICLES,
+        label="Add main subject to scholarly articles and preprints",
+        language_code=SupportedLanguageCode.ENGLISH,
+        best_practice_information=(
+            "When adding QID main subjects please try to first "
+            "educate yourself about the subarea of science a little "
+            "and find/create items as specific as possible.\n"
+            "E.g. when searching for 'cancer screening' in Wikidata "
+            "we find 'gastric cancer screening' in labels of "
+            "scientific articles but there is "
+            "perhaps no item for this yet.\n"
+            "In this case it is preferred to first create that item "
+            "(done in Q108532542 and add that as main subject and "
+            "avoid the more general 'cancer screening' until all "
+            "sub forms of screening have been matched."
+        ),
+        number_of_queries_per_search_string=2
+    ),
+    Task(
+        id=TaskIds.RIKSDAGEN_DOCUMENTS,
+        label="Add main subject to documents from Riksdagen",
+        language_code=SupportedLanguageCode.SWEDISH,
+        best_practice_information=None
+    ),
+    Task(
+        id=TaskIds.THESIS,
+        label="Add main subject to thesis' and technical reports",
+        language_code=SupportedLanguageCode.ENGLISH,
+        best_practice_information=(
+            "When adding QID main subjects please try to first "
+            "educate yourself about the subarea of science a little "
+            "and find/create items as specific as possible.\n"
+            "E.g. when searching for 'cancer screening' in Wikidata "
+            "we find 'gastric cancer screening' in labels of "
+            "scientific articles but there is "
+            "perhaps no item for this yet.\n"
+            "In this case it is preferred to first create that item "
+            "(done in Q108532542 and add that as main subject and "
+            "avoid the more general 'cancer screening' until all "
+            "sub forms of screening have been matched."
+        ),
+        # items=ThesisItems()
+    ),
+]
diff --git a/tasks.py b/tasks.py
deleted file mode 100644
index 9834344..0000000
--- a/tasks.py
+++ /dev/null
@@ -1,25 +0,0 @@
-from helpers.enums import SupportedLanguageCode, TaskIds
-from models.riksdagen_documents import RiksdagenDocumentLabels
-from models.scholarly_articles import ScholarlyArticleLabels
-from models.task import Task
-
-# When adding a new task, also add it in the enum
-
-# scholarly_labels = ScholarlyArticleLabels()
-# riksdagen_labels = RiksdagenDocumentLabels()
-tasks = [
-    Task(
-        id=TaskIds.SCHOLARLY_ARTICLES,
-        label="Add main subject to scholarly articles",
-        question="Is this a valid subject for this article?",
-        labels=ScholarlyArticleLabels(),
-        language_code=SupportedLanguageCode.ENGLISH
-    ),
-    Task(
-        id=TaskIds.RIKSDAGEN_DOCUMENTS,
-        label="Add main subject to documents from Riksdagen",
-        question="Is this a valid subject for this document?",
-        labels=RiksdagenDocumentLabels(),
-        language_code=SupportedLanguageCode.SWEDISH
-    )
-]


Test patch:
diff --git a/tests/test_itemsubjector.py b/tests/test_itemsubjector.py
new file mode 100644
index 0000000..6c82bd3
--- /dev/null
+++ b/tests/test_itemsubjector.py
@@ -0,0 +1,6 @@
+from unittest import TestCase
+
+
+class Test(TestCase):
+    def test_match_existing_main_subjects(self):
+        self.fail()