Package: pysin
PySinPySin is a toolbox for text retrieval in unstructured documents datasets. It contains both a multi-type text extractor and a search engine. To test them, you can use the medical documents generator that is also provided.OS DependenciesYou will needgeckodriverto run the generator. Download it and copy it to your PATH (eg:/usr/local/bin)Debian, Ubuntu, and friendssudoapt-getinstallbuild-essentiallibpoppler-cpp-devpkg-configpython-devFedora, Red Hat, and friendssudoyuminstallgcc-c++pkgconfigpoppler-cpp-develpython-develredhat-rpm-configmacOSbrewinstallpkg-configpopplerConda users may also need libgcc:condainstall-canacondalibgccWindowsCurrently tested only when using conda:Install the Microsoft Visual C++ Build ToolsInstall poppler through conda:condainstall-cconda-forgepopplerInstallpipinstallpysinSearch engineArgumentsThe function search takes 5 arguments.Positionnal arguments :query: your queryinput_path: the path to the directory to search inoutput_path: the path to the directory to put the results inKeyword arguments:scale: can take the valuesrowordocdepending on if the query should be satisfied by a single row or by a whole document. Therowscale is more precise whereas thedocscale is faster. The scale defaults torow.update_cache:Trueto update the cached files (for example if some files have been added to the folder since the last search), elseFalse. Defaults toTrue. If you're working with a huge amount of data that doesn't change, you should setupdate_cachetoFalse.To search the word 'word' within the files of the folder 'path/to/data/' by writing the results in the folder '/path/to/results/', just run the following command :frompysinimportsearchsearch('word','path/to/data/','path/to/results/')QueriesTo search one word beyond multiple ones, just write them side to side in the query.search('word1 word2 word3','path/to/data/','path/to/results/')To search the files where 'mandatory' is and where 'foo' or 'bar' is also (but not necessarily both at the same time), just type the following command :search('+mandatory foo bar','path/to/data/','path/to/results/',scale='doc')The same query holds for therowscale. The previous command might return a document that contains 'mandatory' at the first row and 'foo' at the last one whereas in therowscale, only the occurrences where 'mandatory'AND'foo' (and/or 'bar') are in the same row are returned.To search the rows where 'mandatory' is but 'forbidden' isn't, type the following command :search('mandatory -forbidden','path/to/data/','path/to/results/')To search an expression with several words, use quotes :search('"complex expression"','path/to/data/','path/to/results/')You can obviously combine everything into a single query :search('+mandatory choice1 choic2 "choice3" -"not this one" +"another mandatory"','path/to/data/','path/to/results/')ResultsWhen a research is launched, a folder is made atoutput_pathin which are two files :results.csv: inrowscale, one row correspond to one occurrence and contains the path to the file, the occurrence row number and the context of the occurrence. Indocscale, there are only the paths to the corresponding files.folders.json: returns the number of occurrences in each folder using a tree structureExtractorThe extractor preprocesses all the files to enable the research by converting the handled files into txt cached files. The handled types are csv, doc, docx, html, md, pdf, rtf, txt, xml.To extract all the files within a folder at path 'path', just run :extract('path/to/data')To erase all the cached files, just run :reset_cache('path/to/data')Medical prescriptions generatorThe generator is based on the data of thefakermodule. It can generate both medical prescriptions and medical report. To generate 19 fake medical documents in the folder 'data', just run the following command :generate(19,'path/to/data')Soft modeThe search engine and the extractor can also by used as softs. For the search engine, just run the following command :$pythonsrc/search.py+mandatorychoice1choic2"choice3"-"not this one"+"another mandatory"--input_pathpath/to/data/--output_pathpath/to/results/To search at thedocscale, just add the argument--d.The extractor can be used like this :$pythonsrc/extractor.pypath/to/data/To clear the cached files, just add the argument--reset:$pythonsrc/extractor.py--resetpath/to/data/TrickIf you have to do lots of researchs in one folder, let's sayabsolute/path/to/data/, by putting the results always in the same folder, let's sayabsolute/path/to/results/, and always at the same scale, let's say therowone, then you can create a shortcut to search more easily by running the following commands :$echoaliassearch=\'python/absolute/path/to/search.py--input_path/absolute/path/to/data/--output_path/absolute/path/to/results\'>>~/.bashrc
$source~/.bashrcThen, you're able to do a research from any location by typing :$search+mandatorychoice1choic2"choice3"-"not this one"+"another mandatory"WARNING :before doing this, make sure that thesearchalias doesn't exist yet, for example by running the commandsearchand checking that shell returns the following message :ModuleNotFoundError:Nomodulenamed'apt_pkg'ExampleYou can test this module using theexample.pyscript.TODOmultithreaded researchimprove medication notationnew document typesadapt .doc extraction to windows environmentPublishFirst, you need to havetwineinstalleddpip install --user --upgrade twineMake sure you have bumped the version number insetup.py, then run the following:python setup.py sdist bdist_wheel
python -m twine upload dist/*