Package: gnes
Highlights•Overview•Install•Getting Started•Hub•Documentation•Tutorial•Contributing•Release Notes•BlogWhat is itGNES [jee-nes] isGeneric Neural Elastic Search, a cloud-native semantic search system based on deep neural network.GNES enables large-scale index and semantic search fortext-to-text,image-to-image,video-to-videoandany-to-anycontent form.Highlights💭 To know more about the key tenets of GNES, read this blog post☁️Cloud-Native & Elastic🐣Easy-to-Use🔬State-of-the-ArtGNES isall-in-microservice! Encoder, indexer, preprocessor and router are all running in their own containers. They communicate via versioned APIs and collaborate under the orchestration of Docker Swarm/Kubernetes etc. Scaling, load-balancing, automated recovering, they come off-the-shelf in GNES.How long would it take to deploy a change that involves just switching a layer in VGG? In GNES, this is just one line change in a YAML file. We abstract the encoding and indexing logic to a YAML config, so that you can change or stack encoders and indexers without even touching the codebase.Taking advantage of fast-evolving AI/ML/NLP/CV communities, we learn from best-of-breed deep learning models and plug them into GNES, making sure you always enjoy the state-of-the-art performance.🌌Generic & Universal📦Model as Plugin💯Best PracticeSearching for texts, image or even short-videos? Using Python/C/Java/Go/HTTP as the client? Doesn't matter which content form you have or which language do you use, GNES can handle them all.When built-in models do not meet your requirments, simply build your own withGNES Hub. Pack your model as a docker container and use it as a plugin.We love to learn the best practice from the community, helping our GNES to achieve the next level of availability, resiliency, performance, and durability. If you have any ideas or suggestions, feel free to contribute.OverviewGNES HubGNES Hubship AI/ML models as Docker containers and use Docker containers as plugins. It offers a clean and sustainable way to port external algorithms (with the dependencies) into the GNES framework.GNES Hub is hosted on the Docker Hub.Install GNESThere are two ways to get GNES, either as a Docker image or as a PyPi package.For cloud users, we highly recommend using GNES via Docker.Run GNES as a Docker Containerdockerrungnes/gnes:latest-alpineThis command downloads the latest GNES image (based onAlpine Linux) and runs it in a container. When the container runs, it prints an informational message and exits.💡 Choose the right GNES imageBesides thealpineimage optimized for the space, we also provide Buster (Debian 10.0), Ubuntu 18.04 and Ubuntu 16.04-based images. The table below summarizesall available GNES tags. One can fill in{ver}withlatest,stableorv0..xx.latestrefers to thelatest masterof this repository, whichmay not be stable. We recommend you to use an official release by changing thelatestto a version number, sayv0.0.24, or simply usingstablefor the last release, e.g.gnes:stable-ubuntuTagSize and layersDescription{ver}-alpinebased on Alpine Linux;no deep learning libraries;extremely lightweight and portable, enables fast scaling on even edge devices.{ver}-busterbased on Debian 10.0;no deep learning libraries;recommended for building or extending a GNES-Hub image.{ver}-ubuntu18based on Ubuntu 18.04;no deep learning libraries.{ver}-fullbased on Ubuntu 16.04;python-3.6.8, cuda-10.0, tf1.14, pytorch1.1, faiss, multiple pretrained models;heavy but self-contained, useful in testing GNES end-to-endly.⚠️ Since 2019/10/21, we have stopped hosting the public mirror Tencent Cloud. The old Docker images still exist, but there won't be new images available on Tencent Cloud anymore.We also provide a public mirrorGithub packages. Select the mirror that serves you well.dockerlogin--username=xxxdocker.pkg.github.com/gnes-ai/gnes# login to github package so that we can pull from itdockerrundocker.pkg.github.com/gnes-ai/gnes/gnes:latest-alpineThe table below shows the status of the build pipeline.RegistryBuild statusDocker Hubgnes/gnes:[tag]Github Packagedocker.pkg.github.com/gnes-ai/gnes/gnes:[tag]Install GNES viapipYou can also install GNES as aPython3package via:pipinstallgnesNote that this will only install a "barebone" version of GNES, consists ofthe minimal dependenciesfor running GNES. No third-party pretrained models, deep learning/NLP/CV packages will be installed. We make this setup as the default installation behavior, as a model interested to NLP engineers may not be interested to CV engineers. In GNES, models serve as Docker plugins.🚸 Tensorflow, Pytorch and torchvision are not part of GNES installation. Depending on your model, you may have to install them in advance.Though not recommended, you can install GNES with full dependencies via:pipinstallgnes[all]🍒 Or cherry-picking the dependencies according to the table below: (click to expand...)pip install gnes[bert]bert-serving-server>=1.8.6, bert-serving-client>=1.8.6pip install gnes[flair]flair>=0.4.1pip install gnes[annoy]annoy==1.15.2pip install gnes[chinese]jiebapip install gnes[vision]opencv-python>=4.0.0, imagehash>=4.0pip install gnes[leveldb]plyvel>=1.0.5pip install gnes[test]pylint, memory_profiler>=0.55.0, psutil>=5.6.1, gputil>=1.4.0pip install gnes[transformers]pytorch-transformerspip install gnes[onnx]onnxruntimepip install gnes[audio]librosa>=0.7.0pip install gnes[scipy]scipypip install gnes[nlp]bert-serving-server>=1.8.6, pytorch-transformers, flair>=0.4.1, bert-serving-client>=1.8.6pip install gnes[cn_nlp]pytorch-transformers, bert-serving-client>=1.8.6, bert-serving-server>=1.8.6, jieba, flair>=0.4.1pip install gnes[all]pylint, psutil>=5.6.1, pytorch-transformers, annoy==1.15.2, bert-serving-client>=1.8.6, gputil>=1.4.0, bert-serving-server>=1.8.6, imagehash>=4.0, onnxruntime, memory_profiler>=0.55.0, jieba, flair>=0.4.1, librosa>=0.7.0, scipy, plyvel>=1.0.5, opencv-python>=4.0.0A good way to cherry-pick dependencies is followingthe example in GNES Huband building you own GNES image.Either way, if you end up reading the following message after$ gnesor$ docker run gnes/gnes, then you are ready to go!Getting Started🐣 PreliminariesMicroserviceWorkflowBuilding a flower search engine in 3 minutesDefine the indexing workflowIndexing flower image dataQuerying similar flowersElastic made easyDeploying a flow via Docker Swarm/KubernetesBuilding a cloud-native semantic poem search engine👨‍💻️Take-home messages🐣 PreliminariesBefore we start, let me first introduce two important concepts in GNES:microserviceandworkflow.MicroserviceFor machine learning engineers and data scientists who are not familiar with the concept ofcloud-nativeandmicroservice, one can picture a microservice as an app on your smartphone. Each app runs independently, and an app may cooperate with other apps to accomplish a task. In GNES, we have four fundamental apps, aka. microservices, they are:Preprocessor: transforming a real-world object to a list of workable semantic units;Encoder: representing a semantic unit with vector representation;Indexer: storing the vectors into memory/disk that allows fast-access;Router: forwarding messages between microservices: e.g. batching, mapping, reducing.In GNES, we have implemented dozens of preprocessor, encoder, indexer to process different content forms, such as image, text, video. It is also super easy to plug in your own implementation, which we shall see an example in the sequel.WorkflowNow that we have a bunch of apps, what are we expecting them to do? A typical search system has two fundamental tasks:indexandquery. Index is storing the documents, query is searching the documents. In a neural search system, one may face another task:train, where one fine-tunes an encoder/preprocessor according to the data distribution in order to achieve better search relevance.These three tasks correspond to three differentworkflowsin GNES.Building a flower search engine in 3 minutes📣 Sincev0.0.46GNES Flowhas become the main interface of GNES. GNES Flow provides apythonicandintuitiveway to implement aworkflow, enabling users to run or debug GNES on a local machine. By default, GNES Flow orchestrates all microservices using multi-thread or multi-process backend, it can be also exported to a Docker Swarm/Kubernetes YAML config, allowing one to deliver GNES to the cloud.🔰 The complete example and the corresponding Jupyter Notebookcan be found at here.In this example, we will use the newgnes.flowAPI (gnes >= 0.0.46is required) to build a toy image search system for indexing and retrievingflowersbased on their similarities.Define the indexing workflowLet's first define the indexing workflow by:fromgnes.flowimportFlowflow=(Flow(check_version=False).add_preprocessor(name='prep',yaml_path='yaml/prep.yml').add_encoder(yaml_path='yaml/incep.yml').add_indexer(name='vec_idx',yaml_path='yaml/vec.yml').add_indexer(name='doc_idx',yaml_path='yaml/doc.yml',recv_from='prep').add_router(name='sync',yaml_path='BaseReduceRouter',num_part=2,recv_from=['vec_idx','doc_idx']))Here, we usethe inceptionV4 pretrained modelas the encoder and the built-in indexers for storing vectors and documents. The flow should be quite self-explanatory, if not, you can always convert it to a SVG image and see its visualization:flow.build(backend=None).to_url()Indexing flower image dataTo index our flower data, we need an iterator that generatesbytesstrings and feed thosebytesstrings into the defined flow.defread_flowers(sample_rate=1.0):withtarfile.open('17flowers.tgz')asfp:forminfp.getmembers():ifm.name.endswith('.jpg')andrandom.random()<=sample_rate:yieldfp.extractfile(m).read()We can now do indexing via the multi-process backend:withflow(backend='process')asfl:fl.index(bytes_gen=read_flowers(),batch_size=64)It will take few minutes depending on your machine.Querying similar flowersWe simply sample 20 flower images as queries and search for their top-10 similar images:num_q=20topk=10sample_rate=0.05# do the queryresults=[]withflow.build(backend='process')asfl:forq,rinfl.query(bytes_gen=read_flowers(sample_rate)):q_img=q.search.query.raw_bytesr_imgs=[k.doc.raw_bytesforkinr.search.topk_results]r_scores=[k.score.valueforkinr.search.topk_results]results.append((q_img,r_imgs,r_scores))iflen(results)>num_q:breakHere is the result, where queries are on the first row.Elastic made easyTo increase the number of parallel components in the flow, simply addreplicasto each service:flow=(Flow(check_version=False,ctrl_with_ipc=True).add_preprocessor(name='prep',yaml_path='yaml/prep.yml',replicas=5).add_encoder(yaml_path='yaml/incep.yml',replicas=6).add_indexer(name='vec_idx',yaml_path='yaml/vec.yml').add_indexer(name='doc_idx',yaml_path='yaml/doc.yml',recv_from='prep').add_router(name='sync',yaml_path='BaseReduceRouter',num_part=2,recv_from=['vec_idx','doc_idx']))flow.build(backend=None).to_url()Deploying a flow via Docker Swarm/KubernetesOne can convert aFlowobject to Docker Swarm/Kubernetes YAML compose file very easily via:flow.build(backend=None).to_swarm_yaml()version:'3.4'services:Frontend0:image:gnes/gnes:latest-alpinecommand:frontend --port_in 56086 --port_out 52674 --port_ctrl 49225 --check_versionFalse --ctrl_with_ipc Trueprep:image:gnes/gnes:latest-alpinecommand:preprocess --port_in 52674 --port_out 65461 --host_in Frontend0 --socket_inPULL_CONNECT --socket_out PUB_BIND --port_ctrl 49281 --check_version False --ctrl_with_ipcTrue --yaml_path yaml/prep.ymlEncoder0:image:gnes/gnes:latest-alpinecommand:encode --port_in 65461 --port_out 50488 --host_in prep --socket_in SUB_CONNECT--port_ctrl 62298 --check_version False --ctrl_with_ipc True --yaml_path yaml/incep.ymlvec_idx:image:gnes/gnes:latest-alpinecommand:index --port_in 50488 --port_out 57791 --host_in Encoder0 --host_outsync --socket_in PULL_CONNECT --socket_out PUSH_CONNECT --port_ctrl 58367 --check_versionFalse --ctrl_with_ipc True --yaml_path yaml/vec.ymldoc_idx:image:gnes/gnes:latest-alpinecommand:index --port_in 65461 --port_out 57791 --host_in prep --host_out sync--socket_in SUB_CONNECT --socket_out PUSH_CONNECT --port_ctrl 50333 --check_versionFalse --ctrl_with_ipc True --yaml_path yaml/doc.ymlsync:image:gnes/gnes:latest-alpinecommand:route --port_in 57791 --port_out 56086 --host_out Frontend0 --socket_outPUSH_CONNECT --port_ctrl 51285 --check_version False --ctrl_with_ipc True --yaml_pathBaseReduceRouter --num_part 2To deploy it, simply copy the generated YAML config to a file saymy-gnes.yml, and then dodockerstackdeploy--compose-filemy-gnes.ymlgnes-531Building a cloud-native semantic poem search engineIn this example, we will build a semantic poem search engine using GNES. Unlike the previous flower search example, here we run each service as an isolated Docker container and then orchestrate them via Docker Swarm. It represents a common scenario in the cloud settings. You will learn how to use powerful and customized GNES images fromGNES hub.🔰 Please checkoutthis repository for detailsand follow the instructions to reproduce.👨‍💻️ Take-home messagesLet's make a short recap of what we have learned.GNES isall-in-microservice, there are four fundamental components: preprocessor, encoder, indexer and router.GNES has three typical workflows: train, index, and query.One can leverageGNES Flow APIto define, modify, export or even visualize a workflow.GNES requires an orchestration engine to coordinate all microservices. It supports Kubernetes, Docker Swarm,  or built-in multi-process/thread solution.DocumentationThe official documentation of GNES is hosted ondocs.gnes.ai. It is automatically built, updated and archived on every new release.Tutorial🚧 Tutorial is still under construction. Stay tuned! Meanwhile, we sincerely welcome you to contribute your own learning experience / case study with GNES!How to write your GNES YAML configHow to write a component-wise YAML configModel management with GNES HubPortingPyTorch-Transformersinto GNESPorting multiple modules in a rowCherry-picking dependencies and build your own GNESUnderstanding preprocessor, encoder, indexer and routerIndex and query text data with GNESIndex and query image data with GNESIndex and query video data with GNESUsing GNES with KubernetesUsing GNES in other language (besides Python)Serves HTTP-request with GNES in an end-to-end wayMigrating frombert-as-serviceBenchmarkWe have setupthis repositoryto track the network latency over different GNES versions. As a part of CICD pipeline, this repo gets automatically updated when theGNES masteris updated or a newGNES version is released.Contributing❤️ The beginning is always the hardest. But fear not, even if you find a typo, a missing docstring or unit test, you can simply correct them by making a commit to GNES. Here are the steps:Create a new branch, sayfix-gnes-typo-1Fix/improve the codebaseCommit the changes. Note thecommit message must followthe naming style, sayfix(readme): improve the readability and move sectionsMake a pull request. Note thepull request must followthe naming style. It can simply be one of your commit messages, just copy paste it, e.g.fix(readme): improve the readability and move sectionsSubmit your pull request and wait for all checks passed (usually 10 minutes)Coding styleCommit and PR styles checkAll unit testsRequest reviews from one of the developers from our core team.Get a LGTM 👍 and PR gets merged.Well done! Once a PR gets merged, here are the things happened next:all Docker images tagged with-latestwill be automatically updated in an hour. You may check theits building status at hereon every Friday when a new release is published, PyPi packages and all Docker images tagged with-stablewill be updated accordindly.your contribution and commits will be included inour weekly release note. 🍻More details can be found in thecontributor guidelines.Citing GNESIf you use GNES in an academic paper, you are more than welcome to make a citation. Here are the two ways of citing GNES:\footnote{https://github.com/gnes-ai/gnes}@misc{2019GNES,
  title={GNES: Generic Neural Elastic Search},
  author={Xiao, Han and Yan, Jianfeng and Wang, Feng and Fu, Jie and Liu, Kai},
  howpublished={\url{https://github.com/gnes-ai}},
  year={2019}}LicenseIf you have downloaded a copy of the GNES binary or source code, please note that the GNES binary and source code are both licensed under theGNU Lesser General Public License version 3.