Package: harmonydata
🌐 harmonydata.ac.ukHarmony Python libraryWhat does Harmony do?Psychologists and social scientists often have to match items in different questionnaires, such as "I often feel anxious" and "Feeling nervous, anxious or afraid".This is calledharmonisation.Harmonisation is a time consuming and subjective process.Going through long PDFs of questionnaires and putting the questions into Excel is no fun.EnterHarmony, a tool that usesnatural language processingand generative AI models to help researchers harmonise questionnaire items, even in different languages.Quick start with the codeRead our guide to contributing to Harmony hereor readCONTRIBUTING.md.You can run the walkthrough Python notebook inGoogle Colabwith a single click:You can also download an R markdown notebook to run in R Studio:You can run the walkthrough R notebook in Google Colab with a single click:The Harmony ProjectHarmony is a tool using AI which allows you to compare items from questionnaires and identify similar content. You can try Harmony athttps://harmonydata.ac.uk/appand you can read our blog athttps://harmonydata.ac.uk/blog/.Who to contact?You can contact Harmony team athttps://harmonydata.ac.uk/, or Thomas Wood athttps://fastdatascience.com/.🖥 Installation instructions (video)🖱 Looking to try Harmony in the browser?Visit:https://harmonydata.ac.uk/app/You can also visit our blog athttps://harmonydata.ac.uk/✅ You need Tika if you want to extract instruments from PDFsDownload and install Java if you don't have it already. Download and install Apache Tika and run it on your computerhttps://tika.apache.org/download.htmljava -jar tika-server-standard-2.3.0.jarRequirementsYou need a Windows, Linux or Mac system withPython 3.8 or abovethe requirements inrequirements.txtJava (if you want to extract items from PDFs)Apache Tika(if you want to extract items from PDFs)🖥 Installing Harmony Python packageYou can install fromPyPI.pip install harmonydataLoading all modelsHarmony uses spaCy to help with text extraction from PDFs. spaCy models can be downloaded with the following command in Python:import harmony
harmony.download_models()Matching example instrumentsinstruments = harmony.example_instruments["CES_D English"], harmony.example_instruments["GAD-7 Portuguese"]
questions, similarity, query_similarity, new_vectors_dict = harmony.match_instruments(instruments)How to load a PDF, Excel or Word into an instrumentharmony.load_instruments_from_local_file("gad-7.pdf")Optional environment variablesAs an alternative to downloading models, you can set environment variables so that Harmony calls spaCy on a remote server. This is only necessary if you are making a server deployment of Harmony.HARMONY_SPACY_PATH- determines where model files are stored. Defaults toHOME DIRECTORY/harmonyHARMONY_DATA_PATH- determines where data files are stored. Defaults toHOME DIRECTORY/harmonyHARMONY_NO_PARSING- set to 1 to import a lightweight variant of Harmony which doesn't support PDF parsing.HARMONY_NO_MATCHING- set to 1 to import a lightweight variant of Harmony which doesn't support matching.Loading instruments from PDFsIf you have a local file, you can load it into a list ofInstrumentinstances:from harmony import load_instruments_from_local_file
instruments = load_instruments_from_local_file("gad-7.pdf")Matching instrumentsOnce you have some instruments, you can match them with each other with a call tomatch_instruments.from harmony import match_instruments
all_questions, similarity, query_similarity, new_vectors_dict = match_instruments(instruments)all_questionsis a list of the questions passed to Harmony, in order.similarityis the similarity matrix returned by Harmony.query_similarityis the degree of similarity of each item to an optional query passed as argument tomatch_instruments.⇗⇗ Using a different vectorisation functionHarmony defaults tosentence-transformers/paraphrase-multilingual-MiniLM-L12-v2(HuggingFace link). However you can use other sentence transformers from HuggingFace by setting the environmentHARMONY_SENTENCE_TRANSFORMER_PATHbefore importing Harmony:export HARMONY_SENTENCE_TRANSFORMER_PATH=sentence-transformers/distiluse-base-multilingual-cased-v2Using OpenAI or other LLMs for vectorisationAny word vector representation can be used by Harmony. The below example works for OpenAI'stext-embedding-ada-002model as of July 2023, provided you have create a paid OpenAI account. However, since LLMs are progressing rapidly, we have chosen not to integrate Harmony directly into the OpenAI client libraries, but instead allow you to pass Harmony any vectorisation function of your choice.import openai
import numpy as np
from harmony import match_instruments_with_function, example_instruments
model_name = "text-embedding-ada-002"
def convert_texts_to_vector(texts):
    vectors = openai.Embedding.create(input = texts, model=model_name)['data']
    return [vectors[i]["embedding"] for i in range(len(vectors))]
instruments = example_instruments["CES_D English"], example_instruments["GAD-7 Portuguese"]
all_questions, similarity, query_similarity, new_vectors_dict = match_instruments_with_function(instruments, None, convert_texts_to_vector)💻 Do you want to run Harmony in your browser locally?Download and install Docker:https://docs.docker.com/desktop/install/mac-install/https://docs.docker.com/desktop/install/windows-install/https://docs.docker.com/desktop/install/linux-install/Open a Terminal and rundocker run -p 8000:8000 -p 3000:3000 harmonydata/harmonylocalThen go tohttp://localhost:3000in your browser.Looking for the Harmony API?Visit:https://github.com/harmonydata/harmonyapi📰 The code for training the PDF extraction is here:https://github.com/harmonydata/pdf-questionnaire-extractionDocker imagesIf you are a Docker user, you can run Harmony from a pre-built Docker image.https://hub.docker.com/repository/docker/harmonydata/harmonyapi- just the Harmony APIhttps://hub.docker.com/repository/docker/harmonydata/harmonylocal- Harmony API and React front endContributing to HarmonyIf you'd like to contribute to this project, you can contact us athttps://harmonydata.ac.uk/or make a pull request on ourGithub repository. You can alsoraise an issue.Developing Harmony🧪 Automated testsTest code is intests/folder usingunittest.The testing tooltoxis used in the automation with GitHub Actions CI/CD.Since the PDF extraction also needs Java and Tika installed, you cannot run the unit tests without first installing Java and Tika. See above for instructions.🧪 Use tox locallyInstall tox and run it:pip install tox
toxIn our configuration, tox runs a check of source distribution usingcheck-manifest(which requires your repo to be git-initialized (git init) and added (git add .) at least), setuptools's check, and unit tests using pytest. You don't need to install check-manifest and pytest though, tox will install them in a separate environment.The automated tests are run against several Python versions, but on your machine, you might be using only one version of Python, if that is Python 3.9, then run:tox -e py39Thanks to GitHub Actions' automated process, you don't need to generate distribution files locally.⚙️Continuous integration/deployment to PyPIThis package is based on the templatehttps://pypi.org/project/example-pypi-package/This packageuses GitHub Actions for both testing and publishingis tested when pushingmasterormainbranch, and is published when create a releaseincludes test files in the source distributionusessetup.cfgforversion single-sourcing(setuptools 46.4.0+)⚙️Re-releasing the package manuallyThe code to re-release Harmony on PyPI is as follows:source activate py311
pip install twine
rm -rf dist
python setup.py sdist
twine upload dist/*‎😃💁 Who worked on Harmony?Harmony is a collaboration project betweenUlster University,University College London, theUniversidade Federal de Santa Maria, andFast Data Science.  Harmony is funded byWellcomeas part of theWellcome Data Prize in Mental Health.The core team at Harmony is made up of:Dr Bettina Moltrecht, PhD(UCL)Dr Eoin McElroy(University of Ulster)Dr George Ploubidis(UCL)Dr Mauricio Scopel Hoffmann(Universidade Federal de Santa Maria, Brazil)Thomas Wood(Fast Data Science)📜 LicenseMIT License. Copyright (c) 2023 Ulster University (https://www.ulster.ac.uk)📜 How do I cite Harmony?McElroy, E., Moltrecht, B., Ploubidis, G.B., Scopel Hoffman, M., Wood, T.A., Harmony [Computer software], Version 1.0, accessed athttps://harmonydata.ac.uk/app. Ulster University (2023)