Package: hebsafeharbor
HebSafeHarborA de-identification toolkit for clinical text in Hebrew.HebSafeHarbor was developed according to the requirements described in הצעה למתווה התממה של טקסט רפואי - נמל מבטחים (read morehere)The toolkit integrates and uses open source libraries and assets, includingHebSpacy(that runs NER model based onAlephBERT,NemoandBMC),Presidio, Wikipedia and public lexicons.📝 Demo website is availablehereContentsInstallationGetting startedDocker ComposeServerDemo ApplicationFAQInstallationTo install the package, run the following commands - preferably in a virtual environment# Create conda env (optional)condacreate--namehebsafeharborpython=3.8
condaactivatehebsafeharbor# Install HebSafeHarborpipinstallhebsafeharbor# Download the he_ner_news_trf model used by hebsafeharborpipinstallhttps://github.com/8400TheHealthNetwork/HebSpacy/releases/download/he_ner_news_trf-3.2.1/he_ner_news_trf-3.2.1-py3-none-any.whlAlternatively, you may clone the repo and install all dependencies:# Create conda env (optional)condacreate--namehebsafeharborpython=3.8
condaactivatehebsafeharbor# Install dependenciespipinstall-rrequirements.txt# (Optional) Install package locallypipinstall-e.# Download the he_ner_news_trf model used by hebsafeharborpipinstallhttps://github.com/8400TheHealthNetwork/HebSpacy/releases/download/he_ner_news_trf-3.2.1/he_ner_news_trf-3.2.1-py3-none-any.whlGetting startedPythonfromhebsafeharborimportHebSafeHarborhsh=HebSafeHarbor()text="""שרון לוי התאשפזה ב02.02.2012 וגרה בארלוזרוב 16 רמת גן"""doc={"text":text}output=hsh([doc])print(output[0].anonymized_text.text)# > <שם_> התאשפזה ב<יום_>.02.2012 וגרה <מיקום_> 16 רמת גןDocker ComposeThe easiest way to consume HebSafeHarbor as aservice with a REST APIanddemo applicationis throughdocker-composesetup.Run thedocker-composecommand against thedocker-compose.ymlfile in the root directory to get the latest containers from Docker Hubdocker-composeup-d--buildNavigate in the browser tohttp://server.localhost/docsto access the service swagger.
Navigate in the browser tohttp://demo.localhostto test the demo application.Development modeTo run the containers against the repo's code, run the following command:docker-compose-fdocker-compose-development.ymlup-d--buildServerLocal REST endpointHebSafeHarbor can be consumed as a REST endpoint for the service powered by theFastAPIlibrary.
To run the service locally, you will need to first install the environment requirements as follows:# Create conda env (optional)condacreate--namehebsafeharbor-serverpython=3.8
condaactivatehebsafeharbor-server# Install dependenciespipinstall-rrequirements.txt# Install server dependenciespipinstall-rrequirements-server.txtOnce installation is done, run the following command to initiate the service:uvicornserver:app--reloadThe service should be running athttp://127.0.0.1:8000. Navigate to this URL and make sure you see the following message:{"message":"Welcome to the Hebrew Safe Harbor!"}Now go tohttp://127.0.0.1:8000/docsto see interactive API documentation (Swagger).Alternatively, you can query the service directly by send POST requests tohttp://127.0.0.1:8000/querywith the payload as described in the API documentation.Server DockerTo download and run the official release as a Docker container, run the following commands:# Download imagedockerpullhebsafeharbor/hebsafeharbor# Run containerdockerrun--namehsh_server-d-p8000:8000hebsafeharbor/hebsafeharbor:latestAlternatively, you can build and run the Docker container:# Build imagedockerbuild./--taghebsafeharbor_server# Run containerdockerrun--namehsh_server-d-p8000:8000hebsafeharbor_serverNavigate tohttp://localhost:8000to validate the service is up and running. Similarly, you can go tohttp://localhost:8000/docsandhttp://localhost:8000/query.Demo ApplicationFor experimentation and testing purposes, we provide aStreamlitdemo application.Note that the demo application functions as a (semi-)web client in the sense that it queries the remote REST endpoint and visualizes the results.In order for the demo to work and interact with the server application, you will need to setHSH_SERVERenvironment variable to the REST point URL.For example, assuming you are running the server as docker container:[For Windows]
$env:HSH_SERVER="http://localhost:8000"

[For Linux]
export HSH_SERVER=http://localhost:8000Run locallyTo run the demo locally, first install the requirements as follows:# Create conda env (optional)condacreate--namehebsafeharbor_demopython=3.8
condaactivatehebsafeharbor_demo# Install dependenciespipinstall-rdemo/requirements-streamlit.txtOnce installation is done, run the following command:streamlitrundemo/demo_app.pyNow go tohttp://localhost:8501to access the applicationDemo DockerTo download the official Docker container, run the following commands:# Download imagedockerpullhebsafeharbor/demo_application# Run container - replace <SERVICE URL> with server url (e.g.,docker run -d -e HSH_SERVER=http://172.17.0.2:8000 -p 8501:8501 hebsafeharbor/demo_application:latest)dockerrun--namehsh_demo-d-e"HSH_SERVER=<SERVICE URL>"-p8501:8501hebsafeharbor/demo_application:latestAlternatively, you can build and run the Docker container:# Build imagedockerbuild--namehsh_demodemo/.--taghebsafeharbor_demo# Run container - replace <SERVICE URL> with server url (e.g.,docker run -d -e HSH_SERVER=http://172.17.0.2:8000 -p 8501:8501 hebsafeharbor_demo)dockerrun-d-e"HSH_SERVER=<SERVICE URL>"-p8501:8501hebsafeharbor_demoNavigate tohttp://localhost:8501to access the demo application.FAQWhy does the server contatiner exits with 137 error code?Note that the server container includes a trsnssformer model that consume more than 2gb of memory, increase Docker memory allocation to 6gbRead morehere.What is the server (Docker container) IP address?Run thenetwork inspect bridgecommand to get networking information regarding running dockersdockernetworkinspectbridgeYour response should look like the following:[{"Name":"bridge",..."Containers":{"9611cad28701cfe0877c2bfed9ad2710202492de6d574d42a6714f439cf4f2d2":{"Name":"hsh_server",..."IPv4Address":"172.17.0.2/16",...}},...}]The IP address is underIPv4Addressassociated with"Name": "hsh_server"container underContainers.
In our case,HSH_SERVER = 172.17.0.2Read morehereSpecial ThanksNLP capabilities are based on resources developedg byONLP Lab(the lab git).
EspecialyAlephBERTandNEMO.HebSafeHarbor is an open-source project developed by8400 The Health Network.