Package: encoder-lib
IntroductionModern encoders have more than one stage of encoding. Developers manually create pipes for converting their text in vector.
But some steps are atomic bricks and can be reuseful.
Beside it some encoding takes a lot of time and we reinvent caches.
Encoder library provide simple way initialization and pipeline construction.Installpipinstallencoder-lib[bert_embedded,bert_client]Get startedLet's create bert thin client for bert-as-servicefromencoders.encoder_factoryimportEncoderFactoryencoder_conf_dict={"default":{"type":"bert_client","input_dim":1,"output_dim":768,"params":{"port":5555,"port_out":5556,"ip":"localhost","timeout":5000,}}}encoder_factory=EncoderFactory(encoder_conf_dict)encoder=encoder_factory.get_encoder("default")documents_list=["Hello World!"]vectors=encoder.encode(documents_list)Coll, we have encoder, but each request over network takes time. Let's enhance the encoder and add simple in-memory in cachefromencoders.encoder_factoryimportEncoderFactoryencoder_conf_dict={"default":{"type":"bert_client","input_dim":1,"output_dim":768,"params":{"port":5555,"port_out":5556,"ip":"localhost","timeout":5000,},"cache":{"type":"simple"}}}encoder_factory=EncoderFactory(encoder_conf_dict)encoder=encoder_factory.get_encoder("default")documents_list=["Hello World!"]# Encoder sends request over networkvectors=encoder.encode(documents_list)# This call takes vector from cachevectors=encoder.encode(documents_list)Simple cache stores data in memory without any memory restriction.
Beside it we can keep time on warming up and load pre-computed vectors from file:encoder_conf_dict={"default":{"type":"bert_client","input_dim":1,"output_dim":768,"params":{"port":5555,"port_out":5556,"ip":"localhost","timeout":5000,},"cache":{"type":"simple","params":{"path_desc":{"type":"absolute","file":"/cache/bert_cache.pkl"}}}}}Path objectPath object is flexible description of file location. Current path object version supports:Absolute path - allow to specify full path to filepath_desc:type:absolutefile:full_file_pathRelative path - allow to specify relative path to file.
We separate full file name on two parts relative and base. Relative part is stored in param "file".
Base part is stored in OS environment variable and make you config transferable to other computers.path_desc:type:relativefile:relative_file_nameos_env:ENV_VARExamplespath_desc:type:relativeos_env:BERT_HOMEfile:"cache/bert_cache.pkl"Supported vectorisersBert-as-Service clientBert embeddedTF-IDFComposite vectoriserexample_bert_client:type:bert_clientinput_dim:1output_dim:768params:port:5555port_out:5556ip:localhosttimeout:5000example_bert_embedded:type:bert_embeddedverbose:Trueinput_dim:1output_dim:768params:seq_len:25graph:path_desc:type:relativeos_env:BERT_HOMEfile:model_for_inference.pbtxtvocab:path_desc:type:relativeos_env:BERT_HOMEfile:vocab.txtexample_composite:type:compositeparams:encoders:-example_bert_clientexample_tf_idf:type:tfidfparams:path_desc:type:absolutefile:/dumped_tf_idf/model.pklRelease notes1.3Added parameter seq_len for BERTFeatureExtractor1.2Added parameter verbose for BaseEncoder and all child classesAdded method simple_dump_to_pickle for dumping EncoderCache.1.0Added base functionality for Bert and TF-IDF encoders