Package: pya0
NoteThis branch is for replicating our results in the recent SIGIR paper:One Blade for One Purpose: Advancing Math Information Retrieval using Hybrid Search.The MABOWDOR :guide_dog: training data and checkpoints can be found in ourtraining and inference subdirectory.
Feel free to ask us any questions via email:w32zhong@uwaterloo.caIf you find our work useful, please cite our paper:@inproceedings{zhong2023mabowdor,booktitle={Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval},title={One Blade for One Purpose: Advancing Math Information Retrieval using Hybrid Search},author={Zhong, Wei and Sheng-Chieh Lin and Jheng-Hong Yang and Jimmy Lin},year={2023}}AboutPyA0is a Python wrapper for Approach Zero search engine.
It provides Python interface to make the search engine core easy to play with.In order to build this Python module, you will need to have this repository placed as a git submodule from its parentrepositoryand then issuemakein this directory.Quick StartInstall thePyPI packageusing pip$sudopip3install--upgradepya0If you find pip is unable to find package, update to the latest pip and try again:$sudoapt-getinstallcurlpython3-distutils
$curlhttps://bootstrap.pypa.io/get-pip.py|python3
$sudopipinstall-ihttps://pypi.python.org/simple/--trusted-hostpypi.orgpya0
$python3-c'import pya0'Math token lexingTest a simple math token scanner:importpya0lst=pya0.lex('\\lim_{n\\to\\infty} (1 + 1/n)^n')print(lst)Result:[(269, 'LIM', 'lim'), (274, 'SUBSCRIPT', 'subscript'), (260, 'VAR', "normal`n'"), (270, 'ARROW', 'to'), (260, 'INFTY', 'infty'), (259, 'ONE', "`1'"), (261, 'ADD', 'plus'), (259, 'ONE', "`1'"), (264, 'FRAC', 'frac'), (260, 'VAR', "normal`n'"), (275, 'SUPSCRIPT', 'supscript'), (260, 'VAR', "normal`n'")]Refer totests/directory for more complete example usages.Download prebuilt index and searchSee what prebuilt indexes we have available:python-mpya0--list-prebuilt-indexesntcir-wfbdescriptionNTCIR-12WikipediaFormulaBrowsingimage_filesystemreiserfsmd56e87fc52a8f02c05113034c4b14b3e06urls[https://vault.cs.uwaterloo.ca/s/gySLti89gZF8xz6/download]...One-line command to download the NTCIR-12 WFB index and evaluate its topics:rm-fntcir12_wfb.run
python-mpya0--use-fallback-parser--verbose\--indexntcir-wfb--collectionntcir12-math-browsing-concrete--trec-outputntcir12_wfb.runNow evaluate the generated run:$./eval-ntcir12.shtsv./ntcir12_wfb.run
./ntcir12_wfb.run0.63040.5411Example UsageGenerate NTCIR-12 run:python-mpya0--use-fallback-parser--index../../indexes/mnt-ntcir12_wfb.img/--collectionntcir12-math-browsing-concrete--trec-outputruns/ntcir12_wfb.runGenerate ARQMath (2022) runs:# task 1python-mpya0--stemmerporter--index../../indexes/mnt-arqmath-task1.img/--collectionarqmath-2022-task1-manual--trec-outputruns/arqmath_task1.run# task 2python-mpya0--index../../indexes/mnt-arqmath-task2.img/--collectionarqmath-2022-task2-refined--trec-outputruns/arqmath_task2.runGenerate grid-search runs:python-mpya0--use-fallback-parser--index../../indexes/mnt-ntcir12_wfb.img/--collectionntcir12-math-browsing-concrete--auto-eval./experiments/auto_eval--symbol-scores.tsvEvaluate a run (evaluating ARQMath Task 2 will require downloadingslt_representation_v3):wgethttps://vault.cs.uwaterloo.ca/s/TpSPrZY4xxRYGS2/download-Otraining-and-inference/datasets/latex_representation_v3.zip
unziptraining-and-inference/datasets/latex_representation_v3.zip
./eval-arqmath3/task2/eval.sh--tsv=./slt_representation_v3/--nojudgeTransformer ModelsPlease check out./training-and-inferenceMaking Search Index (optional)First, useutils/corpus_converter.pyto convert raw dataset files tojsonlfile, the latter can be fed toapproach0 indexerdusinga0-crawlers feeder.Datasets can be found here:https://vault.cs.uwaterloo.ca/s/RTJ27g9Ek2kanReWe have made pre-processed jsonl files available, download them using:wgethttps://vault.cs.uwaterloo.ca/s/s2bcWssfAHHyeTF/download-Ontcir12_wfb.jsonl
wgethttps://vault.cs.uwaterloo.ca/s/ANg5XQyGLsZPXLL/download-Oarqmath3_task1.jsonl
wgethttps://vault.cs.uwaterloo.ca/s/tY5SfDgErgkBr28/download-Oarqmath3_task2.jsonlSecond, create index images and mount them as loop devicescd a0-engine
sudo ./indexerd/scripts/vdisk-setup.sh
vdisk-creat.sh reiserfs 1K
vdisk-mount.sh reiserfs vdisk.imgTo make enough space for common datasets, consider these examplesdf -h
/dev/loop6       12G  8.2G  3.9G  68% /home/w32zhong/indexes/mnt-arqmath-task1.img
/dev/loop8       12G  9.0G  3.1G  75% /home/w32zhong/indexes/mnt-arqmath-task2.img
/dev/loop9      5.0G  609M  4.5G  12% /home/w32zhong/indexes/mnt-ntcir12_wfb.imgThrid, run indexerd daemon to create index for NTCIR-12 WFB:cda0-engine/indexerd
./run/indexerd.out-o~/indexes/mnt-ntcir12_wfb.img/-p8935-ethen, run feeder to feed jsonl file to indexerd:cda0-crawlers/feeder
pythonfeeder.py--indexd-urlhttp://localhost:8935/index--bye--corpusntcir12_wfb./feeder.ini~/corpus/ntcir12_wfb.jsonlFor the ARQMath datasets, usearqmath_task1_default__use_porter_stemmerandarqmath_task2_v3as corpus names.Building This PackageBuild for Local PackageBuild and install package locally (for testing):$makeclean
$sudopython3setup.pyinstallthen, you can import as library from system path:importpya0print(dir(pya0))Build for Manylinux DistributionInstall Docker:apt-getupdate
whichdocker||curl-fsSLhttps://get.docker.com-oget-docker.sh
whichdocker||shget-docker.shPull and run imagequay.io/pypa/manylinux_2_24_x86_64at the parent source directory ofapproach0and assume$HOMEis where you put Indri and Jieba code:sudodockerrun-it-v`pwd`:/code-v$HOME:/hostquay.io/pypa/manylinux_2_24_x86_64bashInside docker container, build pya0 as instructed below, so that you have a linux wheel, e.g.,./dist/pya0-0.1-cp35-cp35m-linux_x86_64.whl.Typical build process:# Inside docker, setup system environment...aptupdate
aptinstall-ygitbuild-essentialg++cmakewgetflexbisonpython3
aptinstall-ylibz-devlibevent-devlibopenmpi-devlibxml2-devlibfl-dev
aptinstall-ylibiberty-dev
aptinstall-ybuild-essentialpython-devpython3-pippython3-venv
python3-mpipinstall--upgradebuild# install pip-build tool# Now, start building (or if you enter from the quickstart image)...cd/code
./configure--indri-path=/host/indri--jieba-path=/host/cppjieba(cd/host/indri&&makeclean&&make)# this one takes minutes to buildmakeclean&&makecd./pya0&&makeclean&&makeUsedocker commit $(docker ps -q | head -1) quickstartto save the container for later re-use:sudo docker run -it -v `pwd`:/code -v $HOME:/host quickstart bashCreate apipdistribution package:$rm-rfdistwheelhouse
$python3-mbuildUpload to PyPI.orgEditsetup.pyand bump up version number.Installtwine$aptinstallrustclibssl-devlibffi-dev
$python3-mpipinstall--user--upgradetwineThen inspect the wheel:$auditwheelshow./dist/pya0-*.whl

pya0-0.1-cp35-cp35m-linux_x86_64.whlisconsistentwiththefollowing
platformtag:"linux_x86_64".

Thewheelreferencesexternalversionedsymbolsinthesesystem-
providedsharedlibraries:libgcc_s.so.1withversions{'GCC_3.0'},
libz.so.1withversions{'ZLIB_1.2.0','ZLIB_1.2.3.3','ZLIB_1.2.2.3'},libstdc++.so.6withversions{'GLIBCXX_3.4.10','GLIBCXX_3.4.11','GLIBCXX_3.4.21','GLIBCXX_3.4.15','CXXABI_1.3','CXXABI_1.3.8','GLIBCXX_3.4','CXXABI_1.3.9','GLIBCXX_3.4.9','CXXABI_1.3.1','GLIBCXX_3.4.20'},libpthread.so.0withversions{'GLIBC_2.2.5','GLIBC_2.3.2','GLIBC_2.3.3'},libc.so.6withversions{'GLIBC_2.7','GLIBC_2.17','GLIBC_2.3.4','GLIBC_2.15','GLIBC_2.3','GLIBC_2.3.2','GLIBC_2.4','GLIBC_2.22','GLIBC_2.2.5','GLIBC_2.14'},libdl.so.2withversions{'GLIBC_2.2.5'},libm.so.6
withversions{'GLIBC_2.2.5'},liblzma.so.5withversions{'XZ_5.0'}Thisconstrainstheplatformtagto"manylinux_2_24_x86_64".Inorder
toachieveamorecompatibletag,youwouldneedtorecompileanew
wheelfromsourceonasystemwithearlierversionsofthese
libraries,suchasarecentmanylinuximage.theauditwheelsuggests to use platformmanylinux_2_24_x86_64.Fix it to that platform:$auditwheelrepair./dist/*.whl--platmanylinux_2_24_x86_64-w./wheelhouse
INFO:auditwheel.main_repair:Repairingpya0-0.2.8-py3-none-any.whl
INFO:auditwheel.wheeltools:Previousfilenametags:any
INFO:auditwheel.wheeltools:Newfilenametags:manylinux_2_24_x86_64
INFO:auditwheel.wheeltools:PreviousWHEELinfotags:py3-none-any
INFO:auditwheel.wheeltools:ChangedwheeltypetoPlatlib
INFO:auditwheel.wheeltools:NewWHEELinfotags:py3-none-manylinux_2_24_x86_64
INFO:auditwheel.main_repair:
Fixed-upwheelwrittento/code/pya0/wheelhouse/pya0-0.2.8-py3-none-manylinux_2_24_x86_64.whlThen you should be able to upload to PIP:$python3-mtwineupload--repositorypypiwheelhouse/*.whl(use username__token__and your created token onhttps://pypi.org)Useunzipto view and check if shared libraries are there in the manylinux wheel:root@1c06f5c28b7b:/host/a0-engine/pya0#unzip-lwheelhouse/pya0-0.1.7-py3-none-manylinux_2_24_x86_64.whl
Archive:wheelhouse/pya0-0.1.7-py3-none-manylinux_2_24_x86_64.whlLengthDateTimeName
----------------------------9272021-03-0819:00setup.py20651122021-03-0819:01pya0.libs/libxml2-bbd52ef6.so.2.9.420207362021-03-0819:01pya0.libs/libicuuc-5743fca1.so.57.1432962021-03-0819:01pya0.libs/libltdl-e9c06fbe.so.7.3.12723922021-03-0819:01pya0.libs/libhwloc-811858d2.so.5.7.23122162021-03-0819:01pya0.libs/libevent-2-6d3aa264.0.so.5.1.938050322021-03-0819:01pya0.libs/libicui18n-03536ef3.so.57.11593842021-03-0819:01pya0.libs/liblzma-5b8415cf.so.5.2.26406242021-03-0819:01pya0.libs/libopen-rte-6abe1f34.so.20.1.01086242021-03-0819:01pya0.libs/libz-7fd423a0.so.1.2.810798482021-03-0819:01pya0.libs/libmpi-69c5bc42.so.20.0.27852482021-03-0819:01pya0.libs/libopen-pal-321722b9.so.20.2.0484322021-03-0819:01pya0.libs/libnuma-c8473f23.so.1.0.0256784402021-03-0819:01pya0.libs/libicudata-79cf9efa.so.57.112021-03-0819:01pya0-0.1.7.dist-info/top_level.txt1332021-03-0819:01pya0-0.1.7.dist-info/WHEEL55812021-03-0819:01pya0-0.1.7.dist-info/METADATA17572021-03-0819:01pya0-0.1.7.dist-info/RECORD242021-03-0818:51pya0/__init__.py758784882021-03-0819:01pya0/pya0.so
----------------11290629520files