Package: pprl
PPRL libraryThepprllibrary provides wrappers around the PPRL REST services provided by the Medical Data Science Group Leipzig.
The main entrypoints arepprl.encoder,pprl.matchandpprl.brokerwhich are all submodules for consuming the APIs of the respective services.DocumentationThe documentation of the latest commit on themasterbranchcan be seen on GitLab.Running testsRun the linter in the root directory usingpoetry run flake8.Navigate to thetestsdirectory on the command line and executedocker compose up -d.
This will start a number of services that are required to run the integration tests.
Once they're up and running (might take a couple minutes), run the following command in the root directory of this repository.$ PYTEST_BROKER_BASE_URL="http://localhost:8080/broker" \
    PYTEST_ENCODER_BASE_URL="http://localhost:8080/encoder" \
    PYTEST_MATCH_BASE_URL="http://localhost:8080/matcher" \
    poetry run pytestInstallationRunpip install pprl.
You can then import thepprlmodule in your project.UsageThe following snippet shows how to encode an entity with specific Bloom filter encoding definitions and attribute schemas with theencodersubmodule.
Depending on which parameters you choose, some options may be mandatory, despite them being type hinted as optional.frompprlimportAttributeSchema,BloomFilterConfiguration,Entityfrompprl.encoderimportEncoderClientencoder=EncoderClient("http://localhost:8080/encoder")entities=encoder.encode(config=BloomFilterConfiguration(filter_type="RBF",hash_strategy="RANDOM_SHA256",key="s3cr3t"),schema_list=[AttributeSchema(attribute_name="name",data_type="string",average_token_count=10,weight=2),AttributeSchema(attribute_name="age",data_type="integer",average_token_count=3,weight=1)],entity_list=[Entity(id="1",attributes={"name":"foobar","age":42})])forentityinentities:print(f"{entity.id}={entity.value}")You can use the generated Base64-encoded bit vectors to compute their similarities to one another.
You will need to make use of thematchsubmodule.frompprlimportMatchConfigurationfrompprl.matchimportMatchClientmatcher=MatchClient("http://localhost:8080/matcher")matches=matcher.match(config=MatchConfiguration(match_function="JACCARD",match_mode="CROSSWISE",threshold=0.8),domain_list=["Zm9vYmFyCg=="],range_list=["Zm9vYmF6Cg=="])formatchinmatches:print(f"{match.domain}=>{match.range}({round(match.similarity,3)})")Thebrokersubmodule is for consuming the broker service API.
It is designed for massively parallel distributed record linkage.
As such, the following example is a bit more complicated, but not by much.
Effectively, a new session is created.
Two clients will join the session, submit their bit vectors and receive their results eventually.importtimefrompprlimportBitVector,BitVectorMetadata,BitVectorMetadataSpecification,MatchConfigurationfrompprl.brokerimportBrokerClientbroker=BrokerClient("http://localhost:8080/broker")# we can discard the second argument since we won't receive any cancellation arguments# from the "simple" cancellation strategysession_secret,_=broker.create_session(config=MatchConfiguration(match_function="JACCARD",threshold=0.8),session_cancellation="SIMPLE",metadata_specifications=[BitVectorMetadataSpecification(name="createdAt",data_type="datetime",decision_rule="keepLatest")])# we create two clients identified by different secretsclient_1_secret=broker.create_client(session_secret)client_2_secret=broker.create_client(session_secret)broker.submit_bit_vectors(client_1_secret,[BitVector(id="1",value="Zm9vYmFyCg==",metadata=[BitVectorMetadata(name="createdAt",value="2022-06-21T10:24:36+02:00")])])broker.submit_bit_vectors(client_2_secret,[BitVector(id="2",value="Zm9vYmF6Cg==",metadata=[BitVectorMetadata(name="createdAt",value="2022-06-21T10:25:25+02:00")])])# wait for matching to finish and check back every secondwhilebroker.get_session_progress(session_secret)<1:time.sleep(1)# now print out the results for every clientforclient_secretin(client_1_secret,client_2_secret):print(f"matches for client{client_secret}")formatchinbroker.get_results(client_secret):print(f"{match.vector.id}({round(match.similarity,3)})")# finally, cancel the sessionbroker.cancel_session(session_secret)