Package: text2vec-service
text2vec-serviceBert model to vector service.text2vec-service搭建了一个高效的文本转向量(Text-To-Vector)服务。GuideFeatureInstallUsageContactReferenceFeatureBERT service with C/S.Installpipinstalltorch# conda install pytorchpipinstall-Utext2vec-serviceorpipinstalltorch# conda install pytorchpipinstall-rrequirements.txt

gitclonehttps://github.com/shibing624/text2vec-service.gitcdtext2vec-service
pipinstall--no-deps.Usage1. Start the BERT serviceAfter installing the server, you should be able to useservice-server-startCLI as follows:service-server-start-model_dirshibing624/text2vec-base-chineseThis will start a service with four workers, meaning that it can handle up to fourconcurrentrequests.
More concurrent requests will be queued in a load balancer.Alternatively, one can start the BERT Service in a Docker Container (click to expand...)dockerbuild-ttext2vec-service-f./docker/Dockerfile.NUM_WORKER=1PATH_MODEL=/PATH_TO/_YOUR_MODEL/
dockerrun--runtimenvidia-dit-p5555:5555-p5556:5556-v$PATH_MODEL:/model-ttext2vec-service$NUM_WORKER2. Use Client to Get Sentence EncodesNow you can encode sentences simply as follows:fromservice.clientimportBertClientbc=BertClient()bc.encode(['如何更换花呗绑定银行卡','花呗更改绑定银行卡'])It will return andarray(orList[List[float]]if you wish), in which each row is a fixed-length vector
representing a sentence. Having thousands of sentences? Justencode!Don't even bother to batch,
the server will take care of it.Use BERT Service RemotelyOne may also start the service on one (GPU) machine and call it from another (CPU) machine as follows:# on another CPU machinefromservice.clientimportBertClientbc=BertClient(ip='xx.xx.xx.xx')# ip address of the GPU machinebc.encode(['如何更换花呗绑定银行卡','花呗更改绑定银行卡'])Server and Client API▴ Back to topServer APIservice-server-start--help
service-server-terminate--help
service-server-benchmark--helpArgumentTypeDefaultDescriptionmodel_dirstrRequiredfolder path of the pre-trained BERT model.max_seq_lenint25maximum length of sequence, longer sequence will be trimmed on the right side. Set it to NONE for dynamically using the longest sequence in a (mini)batch.cased_tokenizationboolFalseWhether tokenizer should skip the default lowercasing and accent removal. Should be used for e.g. the multilingual cased pretrained BERT model.num_workerint1number of (GPU/CPU) worker runs BERT model, each works in a separate process.max_batch_sizeint256maximum number of sequences handled by each worker, larger batch will be partitioned into small batches.priority_batch_sizeint16batch smaller than this size will be labeled as high priority, and jumps forward in the job queue to get result fasterportint5555port for pushing data from client to serverport_outint5556port for publishing results from server to clienthttp_portintNoneserver port for receiving HTTP requestscorsstr*setting "Access-Control-Allow-Origin" for HTTP requestsgpu_memory_fractionfloat0.5the fraction of the overall amount of memory that each GPU should be allocated per workercpuboolFalserun on CPU instead of GPUxlaboolFalseenableXLA compilerfor graph optimization (experimental!)fp16boolFalseuse float16 precision (experimental)device_maplist[]specify the list of GPU device ids that will be used (id starts from 0)Client APIArgumentTypeDefaultDescriptionipstrlocalhostIP address of the serverportint5555port for pushing data from client to server,must be consistent with the server side configport_outint5556port for publishing results from server to client,must be consistent with the server side configoutput_fmtstrndarraythe output format of the sentence encodes, either in numpy array or python List[List[float]] (ndarray/list)show_server_configboolFalsewhether to show server configs when first connectedcheck_versionboolTruewhether to force client and server to have the same versionidentitystrNonea UUID that identifies the client, useful in multi-castingtimeoutint-1set the timeout (milliseconds) for receive operation on the clientABertClientimplements the following methods and properties:MethodDescription.encode()Encode a list of strings to a list of vectors.encode_async()Asynchronous encode batches from a generator.fetch()Fetch all encoded vectors from server and return them in a generator, use it with.encode_async()or.encode(blocking=False). Sending order is NOT preserved..fetch_all()Fetch all encoded vectors from server and return them in a list, use it with.encode_async()or.encode(blocking=False). Sending order is preserved..close()Gracefully close the connection between the client and the server.statusGet the client status in JSON format.server_statusGet the server status in JSON format:book: Tutorial▴ Back to topThe full list of examples can be found inexamples/. You can run each viapython examples/base-demo.py.Serving a fine-tuned BERT modelPretrained BERT models often show quite "okayish" performance on many tasks. However, to release the true power of
BERT a fine-tuning on the downstream task (or on domain-specific data) is necessary.In this example, serve a fine-tuned BERT model.service-server-start-model_dirshibing624/bert-base-chineseAsynchronous encodingThe complete example can be foundexamples/async_demo.py.BertClient.encode()offers a nice synchronous way to get sentence encodes.
However, sometimes we want to do it in an asynchronous manner by feeding all textual data to the server first,
fetching the encoded results later. This can be easily done by:# an endless data stream, generating data in an extremely fast speeddeftext_gen():whileTrue:yieldlst_str# yield a batch of text linesbc=BertClient()# get encoded vectorsforjinbc.encode_async(text_gen(),max_num_batch=10):print('received%dx%d'%(j.shape[0],j.shape[1]))Broadcasting to multiple clientsexample:examples/multicast_demo.py.The encoded result is routed to the client according to its identity. If you have multiple clients with
same identity, then they all receive the results! You can use thismulticastfeature to do some cool things,
e.g. training multiple different models (some usingscikit-learnsome usingpytorch) in multiple
separated processes while only callBertServeronce. In the example below,bcand its two clones will
all receive encoded vector.# clone a client by reusing the identitydefclient_clone(id,idx):bc=BertClient(identity=id)forjinbc.listen():print('clone-client-%d: received%dx%d'%(idx,j.shape[0],j.shape[1]))bc=BertClient()# start two cloned clients sharing the same identity as bcforjinrange(2):threading.Thread(target=client_clone,args=(bc.identity,j)).start()for_inrange(3):bc.encode(lst_str)Monitoring the service status in a dashboardThe complete example canbe found in plugin/dashboard/.As a part of the infrastructure, one may also want to monitor the service status and show it in a dashboard. To do that, we can use:bc=BertClient(ip='server_ip')json.dumps(bc.server_status,ensure_ascii=False)This gives the current status of the server including number of requests, number of clients etc. in JSON format. The only thing remained is to start a HTTP server for returning this JSON to the frontend that renders it.Alternatively, one may simply expose an HTTP port when starting a server via:bert-serving-start-http_port8081This will allow one to use javascript orcurlto fetch the server status at port 8081.plugin/dashboard/index.htmlshows a simple dashboard based on Bootstrap and Vue.js.Usingtext2vec-serviceto serve HTTP requests in JSONBesides callingtext2vec-servicefrom Python, one can also call it via HTTP request in JSON. It is quite
useful especially when low transport layer is prohibited. Behind the scene,text2vec-servicespawns a Flask
server in a separate process and then reuse aBertClientinstance as a proxy to communicate with the ventilator.To enable the build-in HTTP server, we need to first (re)install the server with some extra Python dependencies:pipinstall"text2vec-service[http]"Then simply start the server with:bert-serving-start-model_dir=/YOUR_MODEL-http_port8081Done! Your server is now listening HTTP and TCP requests at port8081simultaneously!To send a HTTP request, first prepare the payload in JSON as following:{"id":123,"texts":["hello world","good day!"]}, whereidis a unique identifier helping you to synchronize the results.Then simply call the server at/encodevia HTTP POST request. You can use javascript or whatever, here is an
example usingcurl:curl-XPOSThttp://xx.xx.xx.xx:8081/encode\-H'content-type: application/json'\-d'{"id": 123,"texts": ['如何更换花呗绑定银行卡', '花呗更改绑定银行卡']}', which returns a JSON:{"id":123,"results":[[768float-list],[768float-list]],"status":200}To get the server's status and client's status, you can send GET requests at/status/serverand/status/client,
respectively.Finally, one may also config CORS to restrict the public access of the server by specifying-corswhen
startingbert-serving-start. By default-cors=*, meaning the server is public accessible.StartingBertServerfrom PythonBesides shell, one can also start aBertServerfrom python. Simply dofromservice.server.helperimportget_args_parserfromservice.serverimportBertServerargs=get_args_parser().parse_args(['-model_dir','YOUR_MODEL_PATH_HERE','-port','5555','-port_out','5556','-max_seq_len','NONE','-mask_cls_sep','-cpu'])server=BertServer(args)server.start()Note that it's basically mirroring the arg-parsing behavior in CLI, so everything in that.parse_args([])list
should be string, e.g.['-port', '5555']not['-port', 5555].To shutdown the server, you may call the static method inBertServerclass via with args:shut_args=get_shutdown_parser().parse_args(['-ip','localhost','-port','5555','-timeout','5000'])BertServer.shutdown(shutdown_args)Or via shell CLI:bert-serving-terminate-port5555This will terminate the server running on localhost at port 5555. You may also use it to terminate a remote server,
seebert-serving-terminate --helpfor details.ContactIssue(建议)：邮件我：xuming:xuming624@qq.com微信我：
加我微信号：xuming624, 备注：姓名-公司-NLP进NLP交流群。Citation如果你在研究中使用了text2vec-service，请按如下格式引用：APA:Xu, M. text2vec-service: Bert model embedding service (Version 0.0.2) [Computer software]. https://github.com/shibing624/text2vec-serviceBibTeX:@software{Xu_text2vec-service_Text_to,
author ={Xu, Ming},
title ={{text2vec-service: Bert model embedding service}},
url ={https://github.com/shibing624/text2vec-service},
version ={0.0.2}}License授权协议为The Apache License 2.0，可免费用做商业用途。请在产品说明中附加text2vec-service的链接和授权协议。Contribute项目代码还很粗糙，如果大家对代码有所改进，欢迎提交回本项目，在提交之前，注意以下两点：在tests添加相应的单元测试使用python -m pytest -v来运行所有单元测试，确保所有单测都是通过的之后即可提交PR。Referencejina-ai/clip-as-servicehuggingface/transformers