Package: sentence-embedding-evaluation-german
sentence-embedding-evaluation-germanSentence embedding evaluation for German.This library is inspired bySentEvalbut focuses on German language downstream tasks.Downstream tasksThe available downstream tasks are listed in the table below.
If you that think that a dataset is missing and should be added, pleaseopen an issue.tasktypetext typelang#train#testtargetinfoTOXIC👿 toxic commentsfacebook commentsde-DE3244944binary {0,1}GermEval 2021, comments subtask 1,📁📖ENGAGE🤗 engaging commentsfacebook commentsde-DE3244944binary {0,1}GermEval 2021, comments subtask 2,📁📖FCLAIM☝️ fact-claiming commentsfacebook commentsde-DE3244944binary {0,1}GermEval 2021, comments subtask 3,📁📖VMWE☁️ verbal idiomsnewspaperde-DE66521447binary (figuratively, literally)GermEval 2021, verbal idioms,📁📖OL19-A👿 offensive languagetweetsde-DE39803031binary {0,1}GermEval 2018,📁📖OL19-B👿 offensive language, fine-grainedtweetsde-DE398030314 catg. (profanity, insult, abuse, oth.)GermEval 2018,📁📖OL19-C👿 explicit vs. implicit offensetweetsde-DE1921930binary (explicit, implicit)GermEval 2018,📁📖OL18-A👿 offensive languagetweetsde-DE50093398binary {0,1}GermEval 2018,📁OL18-B👿 offensive language, fine-grainedtweetsde-DE500933984 catg. (profanity, insult, abuse, oth.)GermEval 2018,📁ABSD-1🤷 relevance classification'Deutsche Bahn' customer feedbackde-DE194322555binaryGermEval 2017,📁ABSD-2😃😐😡 sentiment analysis'Deutsche Bahn' customer feedbackde-DE1943225553 catg. (pos., neg., neutral)GermEval 2017,📁ABSD-3🛤️ aspect categories'Deutsche Bahn' customer feedbackde-DE19432255520 catg.GermEval 2017,📁MIO-S😃😐😡 sentiment analysis'Der Standard' newspaper article web commentsde-AT179918003 catg.One Million Posts Corpus,📁MIO-O🤷 off-topic comments'Der Standard' newspaper article web commentsde-AT17991800binaryOne Million Posts Corpus,📁MIO-I👿 inappropriate comments'Der Standard' newspaper article web commentsde-AT17991800binaryOne Million Posts Corpus,📁MIO-D👿 discriminating comments'Der Standard' newspaper article web commentsde-AT17991800binaryOne Million Posts Corpus,📁MIO-F💡 feedback comments'Der Standard' newspaper article web commentsde-AT30193019binaryOne Million Posts Corpus,📁MIO-P✉️ personal story comments'Der Standard' newspaper article web commentsde-AT46684668binaryOne Million Posts Corpus,📁MIO-A✴️ argumentative comments'Der Standard' newspaper article web commentsde-AT17991800binaryOne Million Posts Corpus,📁SBCH-S😃😐😡 sentiment analysis'chatmania' app comments, only comments labelled as Swiss German are includedgsw3943943 catg.SB-CH Corpus,📁SBCH-L⛰️ dialect classification'chatmania' app commentsgsw748748binarySB-CH Corpus,📁ARCHI⛰️ dialect classificationAudio transcriptions of interviews in four dialect regions of Switzerlandgsw1880947434 catg.ArchiMob,📁📖LSDC🌊 dialect classificationseveral genres (e.g. formal texts, fairytales, novels, poetry, theatre plays) from the 19th to 21st centuries. Extincted Lower Prussia excluded. Gronings excluded due to lack of test examples.nds74140860214 catg.Lower Saxon Dialect Classification,📁📖KLEX-P🤔 text levelConceptual complexity classification of texts written for adults (Wikipedia), children between 6-12 (Klexikon), and beginner readers (MiniKlexikon); Paragraph split indicated by<eop>or*de826481533 catg.📁📖Download datasetsbashdownload-datasets.shCheck if files were actually downloadedfind./datasets/**/-execls-lh{}\;Usage exampleImport the required Python packages.fromtypingimportListimportsentence_embedding_evaluation_germanasseegimporttorchStep (1) Load your pretrained modelIn the following example, we generate a random embedding matrix for demonstration purposes.# (1) Instantiate an embedding modelemb_dim=512vocab_sz=128emb=torch.randn((vocab_sz,emb_dim),requires_grad=False)emb=torch.nn.Embedding.from_pretrained(emb)assertemb.weight.requires_grad==FalseStep (2) Specify yourpreprocessorfunctionYou need to specify your own preprocessing routine.
Thepreprocessorfunction must convert a list of stringsbatch(List[str])
into a list of feature vectors, or resp. a list of sentence embeddings (List[List[float]]).
In the following example, we generate some sort of token IDs, retrieve the vectors from our random matrix, and average these to feature vectors for demonstration purposes.# (2) Specify the preprocessingdefpreprocesser(batch:List[str],params:dict=None)->List[List[float]]:""" Specify your embedding or pretrained encoder hereParamters:----------batch : List[str]A list of sentence as stringparams : dictThe params dictionaryReturns:--------List[List[float]]A list of embedding vectors"""features=[]forsentinbatch:try:ids=torch.tensor([ord(c)%128forcinsent])except:print(sent)h=emb(ids)features.append(h.mean(axis=0))features=torch.stack(features,dim=0)returnfeaturesStep (3) Training settingsWe suggest to train a final layer with bias term ('bias':True),
on a loss function weighted by the class frequency ('balanced':True),
a batch size of 128, an over 500 epochs without early stopping.# (3) Training settingsparams={'datafolder':'./datasets','bias':True,'balanced':True,'batch_size':128,'num_epochs':500,# 'early_stopping': True,# 'split_ratio': 0.2,  # if early_stopping=True# 'patience': 5,  # if early_stopping=True}Step (4) Downstream tasksWe suggest to run the following downstream tasks.FCLAIMflags comments that requires manual fact-checking because these contain reasoning, arguments or claims that might be false.VMWEdifferentiates texts with figurative or literal multi-word expressions.OL19-Cdistincts between explicit and implicit offensive language.ABSD-2is a sentiment analysis dataset with customer reviews.
These four dataset so far can be assumed to be Standard German from Germany (de-DE).MIO-Pflags Austrian German (de-AT) comments if these contain personal stories.ARCHIis a Swiss (gsw), andLSDCa Lower German (nds) dialect identification task.# (4) Specify downstream tasksdownstream_tasks=['FCLAIM','VMWE','OL19-C','ABSD-2','MIO-P','ARCHI','LSDC']Step (5) Run the experimentsFinally, start the evaluation.
The suggested downstream tasks (step 4) with 500 epochs (step 3)
might requires 10-40 minutes but it's highly dependent on your computing resources.
So grab a ☕ or 🍵.# (5) Run experimentsresults=seeg.evaluate(downstream_tasks,preprocesser,**params)Demo notebooksStart Jupytersource.venv/bin/activate
jupyterlabOpen an demo notebookGeneric demodeepset examplefasttext exampleSBert exampleAppendixInstallation & DownloadsThesentence-embedding-evaluation-germangit repois available asPyPi packagepipinstallsentence-embedding-evaluation-german
pipinstallgit+ssh://git@github.com/ulf1/sentence-embedding-evaluation-german.gitYou need to download the datasets as well.
If you run the following code, the datasets should be in a folder./datasets.wget-q"https://raw.githubusercontent.com/ulf1/sentence-embedding-evaluation-german/main/download-datasets.sh"-Odownload-datasets.shbashdownload-datasets.shDevelopment work for this packageInstall a virtual environmentpython3-mvenv.venvsource.venv/bin/activate
pipinstall--upgradepip
pipinstall-rrequirements.txt--no-cache-dir
pipinstall-rrequirements-dev.txt--no-cache-dir
pipinstall-rrequirements-demo.txt--no-cache-dir(If your git repo is stored in a folder with whitespaces, then don't use the subfolder.venv. Use an absolute path without whitespaces.)Install conda environment for GPUcondainstall-ypip
condacreate-y--namegpu-venv-seegpython=3.9pip
condaactivategpu-venv-seeg# install CUDA supportcondainstall-ycudatoolkit=11.3.1cudnn=8.3.2-cconda-forgeexportLD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
pipinstalltorch==1.12.1+cu113torchvisiontorchaudio-fhttps://download.pytorch.org/whl/torch_stable.html# install other packagespipinstall-rrequirements.txt--no-cache-dir
pipinstall-rrequirements-dev.txt--no-cache-dir
pipinstall-rrequirements-demo.txt--no-cache-dir
watch-n0.5nvidia-smiPython commandsJupyter for the examples:jupyter labCheck syntax:flake8 --ignore=F401 --exclude=$(grep -v '^#' .gitignore | xargs | sed -e 's/ /,/g')Publish packagepandocREADME.md--frommarkdown--torst-s-oREADME.rst
pythonsetup.pysdisttwineupload-rpypidist/*Clean upfind.-typef-name"*.pyc"|xargsrm
find.-typed-name"__pycache__"|xargsrm-r
rm-r.pytest_cache
rm-r.venvNew Dataset recommendationIf you want to recommend another or a new dataset, pleaseopen an issue.TroubleshootingIf you have troubles to get this package running, pleaseopen an issuefor support.ContributingPlease contribute usingGithub Flow. Create a branch, add commits, andopen a pull request.CitationIf you want to use this package in a research paper, pleaseopen an issuebecause we have not yet decided how to make this package citable. You should at least mention the PyPi version in your paper to ensure reproducibility.You certainly need to cite the actual evaluation datasets in your paper. Please check the hyperlinks in the info column of thetable above.