Package: osmanager
:rocket: osmanSTRVrepository with useful functions to ease (y)our
work with OpenSearch.osmanstands for OpenSearch MANager.:mag: ContentInstallationUsageContributionLocal environment setupTestingLocal linteringVersioningContributersOpenSearchSpecs:computer: InstallationCreate a new virtual environment.Runpip install osmanager.:hammer: UsageCreate an Osman instanceTheenvironment variablesare read byOsmanConfig.from osman import Osman, OsmanConfig

os_man = Osman(OsmanConfig())Environment variables can be overridden.os_man = Osman(OsmanConfig(host_url=<OpenSearch_host_url>))Create an indexmapping = {
  "mappings": {
    "properties": {
      "age": {"type": "integer"},
      "id": {"type": "integer"},
      "name": {"type": "text"},
    }
  }
}

settings = {
  "settings": {
    "number_of_shards": 3
  }
}

os_man.create_index(
  name=<index_name>, mapping=mapping, settings=settings
)Upload a search templatesource = {
  "query": {
    "match": {
      "age": "{{age}}"
    }
  }
}

params = {
  "age": 10
}

os_man.upload_search_template(
            source=source, name=<template_name>, index=<index_name>, params=params
)Upload a painless scriptsource = "doc['first.keyword'].value + ' ' + doc['last.keyword'].value"

os_man.upload_painless_script(source=source, name=<script_name>)Delete a search template or a painless scriptRemoves either a painless script or a search template.os_man.delete_script(name=<script_or_template_name>)Debug a painless scriptExecutes a certain painless script with provided data and parameters. It then checks if the expected result is returned. Thecontext_typemust be provided and is eitherscoreorfilter. This refers to the score or filter queries as is describedhere.context_type = "score"

documents = {"id": 1, "container": [1, 2, 3]}

expected_result = 0

os_man.debug_painless_script(
  source=<source>,
  index=<index_name>,
  params=<params>,
  context_type=context_type,
  documents=documents,
  expected_result=expected_result
)Debug a search templateExecutes a certain search template against an index with defined parameters. It then checks if the expected indices are returned.expected_ids = ["123", "10"]

os_man.debug_search_template(
  source=source, name=<template_name>, index=<index_name>, params=params, expected_ids=expected_ids
)ReindexReindex an existing index with a new mapping and/or settings.
In order to reindex, this function adds a suffix [1, 2] to the index name.
Afterwards, the index should be referenced by its alias rather than its name.For example:An index with the nametest-indexis reindexed. Its name becomestest-index-1.  When reindexed again, its name will becometest-index-2. Hence, it should be referenced by its unchanging aliastest-index.os_man.reindex(
  name=<index_name>, mapping=<new_mapping>, settings=<new_settings>
):construction_worker_man: Contribution:wrench: Local environment setupLaunch the docker daemon in advance of the following steps. You can develop the code using
virtualenv but the local OpenSearch instance requires docker.Local OpenSearch instance and docker development environmentCommandmake helpshows a brief description of possible targets. A typical workflow scenario is:Runmake docker-run-opensearchto launch OpenSearch instance
on port 9200 and OpenSearch Dashboards on port 5601, as described indocker-compose-opensearch.yml.Runmake dev-envto get a bash shell in the development environment defined
inDockerfile. Under Linux set the user/group in advance in.env, see bellow.Develop.Runmake docker-clean-allto clean unused containers and volumes.Note the following:Under Linux you may encounter a problem that a user in the docker guest container
doesn't have permissions to write to the local directory. This leads to problems withpytestunable to create cache directories. The solution is to have the followig
variables in.envfile:DEV_USER_ID=1000
  DEV_GROUP_ID=1000Substitute 1000 byuidandgid(introduction inthis article)
of the user on a host machine. The ids can be obtained by runningidcommand.
Seeexplanationfor why we do that. With Docker under MacOS or Windows you won't need this.You can browse the Dashboards from the web browser. In case
of running Docker on localhost you can browse the Dashboards onhttp://localhost:5061.All indexed data and dashboards are persistent in a docker volume,
i.e. when you stop the OpenSearch containers the data arenotlost.In order to run notebook inside the docker container, use the following command
(and ensurenotebookis in your dependencies):jupyter notebook --ip 0.0.0.0 --no-browser --allow-rootvirtualenvCreate virtual environment calledvenv:virtualenv --python=python3.8 venvActivate it:. ./venv/bin/activateInstall python package:pip install -e .In order to deactivate the environment, rundeactivatecommand.You can also delete the environment as following:rm -r ./venv/:traffic_light: TestingRunpytestin your devel environment to run all tests.TheOsmanConfigclass can be initialized from the environment by the following variables:VariableDefault valueTypeDescriptionAUTH_METHODhttpstringhttpfor username/password authentication,awsauthfor authentication with AWS user credentialsOPENSEARCH_HOSTNonestringaddress of OpenSearch hostOPENSEARCH_PORT443intport numberOPENSEARCH_SSL_ENABLEDTruebooluse SSL?OPENSEARCH_USERNonestringusername, forhttpAUTH_METHODOPENSEARCH_SECRETNonestringpassword, forhttpAUTH_METHODAWS_ACCESS_KEY_IDNonestringaccess key id forawsauthAUTH_METHOD, seeAWS4AuthAWS_SECRET_ACCESS_KEYNonestringsecret key forawsauthAUTH_METHODAWS_REGIONus-east-1stringAWS region forawsauthAUTH_METHODAWS_SERVICEesstringAWS service forawsauthAUTH_METHODYou can add these variables to your.envfile,make dev-envwill pass
them to the devel Docker image. There is a test intest_osman.pycreatingOsmaninstance
using environment variables so you can use any OpenSearch instance for testing.:broom: Local linteringFor running linters from GitHub actions locally, you need to do the following.Installpre-commitlibrary.From root project directory, run:pre-commit run --all-files:heavy_plus_sign: VersioningFor information on semantic versioning, seesemver.org.Given a version number MAJOR.MINOR.PATCH, increment the:MAJOR version when you make incompatible API changesMINOR version when you add functionality in a backwards compatible mannerPATCH version when you make backwards compatible bug fixesWhen incrementing the MAJOR version, reset the MINOR and PATCH versions to 0.
When incrementing the MINOR version, reset the PATCH version to 0.When a version is released, a tag should be created in the formatvMAJOR.MINOR.PATCH.Follow the steps below to create a new release:Updatepackage.jsonwith the new version number.Add the tag to the current branch like this:git tag -a v1.0.0 -m "Release version 1.0.0"Push the tag to the remote repository:git push origin --tagsCreate a new pull request with the new version number and merge it to themasterbranch.:pencil: ContributorsJaroslav BezdekNiek MereuVladimír Kadlec:scroll: OpenSearchThis section contains a brief overview of OpenSearch. This section is by no means complete or exhaustive. However, it contains a few tips and tricks that might be useful for the development of your OpenSearch project. For more information, visit theOpenSearch documentation.:clipboard: AnalyzersAnalyzers are used to process text fields during indexing and search time. Analyzers are composed of one or more tokenizer and zero or more token filters. The tokenizer breaks the text into individual terms and the token filters potentially remove them from the analyzed field. Fields can have more than one field type, so different analyzers can be utilised for different situations.This section contains a small overview of the analyzers available in OpenSearch. For analyzers in general, seeAnalyzers in OpenSearch.StandardStandard analyzers are used to index and search for complete words. Standard analyzers are composed of a tokenizer and a token filter. The standard analyzer uses the standard tokenizer. The standard tokenizer breaks text into terms on word boundaries. The standard analyzer also uses the lowercase token filter. The lowercase token filter converts all tokens to lowercase.The standard tokenizer breaks down words on word boundaries. It breaks words down on whitespaces, as well as many special characters.The Standard analyzer might not be useful for fields like Email addresses. An email address like "123@strv-2.com" is broken down into "123", "strv", "2", "com".WhitespaceWhitespace analyzers are used to index and search for complete words. Whitespace analyzers are composed of a tokenizer and a token filter. The whitespace analyzer uses the whitespace tokenizer.Because the Whitespace analyzer does not break words on special characters, it is more suitable for data like Email addresses or URLs.N-gramN-gram analyzers are used to index and search for partial words. N-gram analyzers are composed of a tokenizer and a token filter. The tokenizer breaks the text into individual terms and the token filter creates n-grams for each term. N-gram analyzers are used to index and search for partial words.N-gram analyzers should not be used for large text fields. This is because n-grams can be very large and can consume a lot of disk space.Words lose their meaning when they are broken into n-grams. For example, the word "search" is broken into "se", "ea", "ar", "rc", "ch". This means that a search for "sea" will match the word "search".N-gram analyzers could be useful when adressing a "username" field. This is because usernames are of limited length and users should probably be found by just searching for the middle part of their name. For example "xSuperUser" should be found by using "sup".Not analysedStandardWhitespaceN-gram123@strv-2.com[123, strv, 2, com][123@strv-2.com][1, 12, 2, 23, 3...]How's it going?[How, s, it, going][How's, it, going?][H, Ho, o, ow, w...]xUser-123[xUser, 123][xUser-123][x, xu, u, us, s...]:card_index: Field TypesThis section contains a small overview of the field types that are available in OpenSearch. For field types that are not discussed here, please refer to theofficial documentation.KeywordKeyword fields are used to store data that is not meant to be analyzed. Keyword fields are not analyzed and support exact value searches.Keyword fields are not used for full-text search.Keyword fields are for example useful for "id" fields.TextText fields are used to store textual data, such as the body of an email or the description of a product in an e-commerce store. Text fields are analyzed and support full-text search.Text fields are used when exact value searches are not wanted.:books: SpecsRequirementsLibraryThe goal is to extendopensearch-py libraryfeatures that we found useful.Connect to opensearch using url (user/pass) or service accountUpload & remove search templateUpload & remove functionCreate & drop index with mappingLoad json file with template source elementLoad json file with functionLoad json file with mappingRun local search template with sample parameters to see if all work (without upload)Run local function with sample parameters (without upload)Run local index with sample doc with given data types (without upload)Check local file vs opensearch file and show differencessearch templatefunctionindex mappingApplicationThe goal is to allow teams manage opensearch instances easily with templated project setup.Have yaml config, yaml list of filesCreate sample json structure for all filesSync things based on yaml file:sync between local json files and yaml listsync between local yaml list and opensearchSplit everything between envs: have yaml file with env definitionsNice to havelint template?vscode default json linter fails to lint template due to parameters syntax: {{#bla}}TasksTasks are kept in github projectOsman