Package: planchet
PlanchetYour large data processing personal assistantAboutPlanchet (pronounced /plʌ̃ʃɛ/) is a data package manager suited for processing large arrays of data
items. It supports natively reading and writing into CSV and JSONL data files
and serving their content over a FastAPI service to clients that process the
data. It is a tool for scientists and hackers, not production.How it worksPlanchet solves the controlled processing of large amounts of data in a simple
and slightly naive way by controlling the reading and writing of the data as
opposed to the processing. When you create a job with Planchet you tell the
sevice where to read, where to write and what classes to use for that. Next,
you (using the client or simple HTTP requests) ask the service forndata
items, which your process works through locally. When your processing is done,
it ships the items back to Planchet, who writes them to disk. All jobs and
serving and receiving of items is logged in a Redis instance with persistence.
This assures that if you stop processing you will only lose the processing of
the data that was not sent back to Planchet. Planchet will automatically resume
jobs and skip over processed items.Caveat:Planchet is running in a single thread to avoid the mess of multiple
processes writing in the same file. Until this is fixed (may be never) you
should be careful with your batch sizes -- keep them not too big and not too
small.Read more about planchet on thedocumentation page.InstallationPlanchet works in two components: a service and a client. The service is the
core that does all the work managing the data while the client is a light
wrapper aroundrequeststhat makes it easy to access the service API.ServiceYou can use this repo and start streight away like this:gitclonegit@github.com:savkov/planchet.gitexportPLANCHET_REDIS_PWD=<some-password>
makeinstall
makerun-redis
makerunIf you want to run Planchet on a different port, you can use theuvicorncommand but note that youMUSTuse only one worker.uvicornapp:app--reload--host0.0.0.0--port5005--workers1You can also run docker-compose from the git repo:gitclonegit@github.com:savkov/planchet.gitexportPLANCHET_REDIS_PWD=<some-password>
docker-composeupClientpipinstallplanchetExampleOn the serverOn the server we need to install Planchet and download some news headlines data
in an accessible directory. Then we multiply the data 1000 times as there are
only 200 lines originally. Don't forget to set yourRedis passwordbefore
you domake install-redis!gitclonehttps://github.com/savkov/planchet.gitcdplanchet
mkdirdata
wgethttps://raw.githubusercontent.com/explosion/prodigy-recipes/master/example-datasets/news_headlines.jsonl-Odata/news_headlines.jsonl
python-c"news=open('data/news_headlines.jsonl').read();open('data/news_headlines.jsonl', 'w').write(''.join([news for _ in range(200)]))"exportPLANCHET_REDIS_PWD=<your-redis-password>
makeinstall
makeinstall-redis
makerunNote that planchet will run at port 5005 on your host machine.On the clientOn the client side we need to install the Planchet client andspaCy.pipinstallplanchetspacytqdm
python-mspacydownloaden_core_web_smexportPLANCHET_REDIS_PWD=<your-redis-password>Then we write the following script in a file calledspacy_ner.pymaking sure
you fill in the placeholders.fromplanchetimportPlanchetClientimportspacyfromtqdmimporttqdmnlp=spacy.load("en_core_web_sm")PLANCHET_HOST='localhost'# <--- CHANGE IF NEEDEDPLANCHET_PORT=5005url=f'http://{PLANCHET_HOST}:{PLANCHET_PORT}'client=PlanchetClient(url)job_name='spacy-ner-job'metadata={# NOTE: this assumes planchet has access to this path'input_file_path':'./data/news_headlines.jsonl','output_file_path':'./data/entities.jsonl'}# make sure you don't use the clean_start option hereclient.start_job(job_name,metadata,'JsonlReader',writer_name='JsonlWriter')# make sure the number of items is large enough to avoid blocking the servern_items=100headlines=client.get(job_name,n_items)whileheadlines:ents=[]print('Processing headlines batch...')forid_,itemintqdm(headlines):item['ents']=[ent.textforentinnlp(item['text']).ents]ents.append((id_,item))client.send(job_name,ents)headlines=client.get(job_name,n_items)Finally, we want to do some parallel processing with 8 processes. We can start
each process manually or we can use theparalleltool to start them all.seq-w08|parallelpythonspacy_ner.py{}ContributorsSasho SavkovDilyan G.Kristian Boda