Package: es2loki
es2lokies2lokiis a migration library that helps to transfer logs from
Elasticsearch to Grafana Loki.To use es2loki currently you need to define your own mapping of elasticsearch documents
to labels for Grafana Loki.DemoYou may find helpful ademofolder which contains a fully-sufficient demo stand
that demonstrates transferring logs usinges2loki.UsageIn the simplest form you don't need to write any Python code at all,
Loki will receive no meaningful labels, but nevertheless - let's see how it works.$pipinstall-Ues2loki
$ELASTIC_HOSTS=http://localhost:9200\ELASTIC_INDEX="filebeat-*"\LOKI_URL=http://localhost:3100\python-mes2lokiIn order to override defaultes2lokibehaviour you need to subclass
aes2loki.BaseTransferclass.To declare how documents map to Loki labels you have to override aextract_doc_labelsmethod (seedemo/example.py):fromes2lokiimportBaseTransferclassTransferLogs(BaseTransfer):defextract_doc_labels(self,source:dict)->Optional[MutableMapping[str,str]]:returndict(app=source.get("fields",{}).get("service_name"),job="logs",level=source.get("level"),node_name=source.get("host",{}).get("name"),logger_name=source.get("logger_name"),)You can run this using the following code:importsysfromes2lokiimportrun_transferif__name__=="__main__":sys.exit(run_transfer(TransferLogs()))You can find more examples in thedemofolder.SortingBy defaultes2lokiassumes that in the documents returned from Elasticsearch
there are fields@timestamp(you can change the name - see below) andlog.offset.
Using these 2 fields we can be sure that we will not reread the same lines multiple times.
But if you have your fields that could guarantee such a behaviour - please
override amake_es_sortandmake_es_search_aftermethods.make_es_sortdefines by which fields the sorting will happen.make_es_search_afterdefines an initial "offset". It is needed to resume es2loki after a shutdown. By default it
extracts information from the internal state, which can be saved persistently.Persistencees2lokihas a mechanism to store the Elasticsearch scrolling state
in the database (highly recommended). In this modees2lokisaves
the scrolling state inside an SQL database (PostgreSQL, MySQL, SQLite, ...).You can opt out of enabling persistence completely usingSTATE_MODE=noneenv variable, which is the default.
But we highly recommend to enable persistence with some SQL storage.DeploymentYou can deployes2lokivia our helm chart.Addktsrepo:helmrepoaddktshttps://charts.kts.studio
helmrepoupdateInstall the chart:helmupgrade--installRELEASE_NAMEkts/es2lokiMore information about helm chart deployment can be foundhere.ConfigurationYou can configurees2lokiusing the following environment variables:namedefaultdescriptionELASTIC_HOSTShttp://localhost:9200Elasticsearch hosts. Separate multiple hosts using,ELASTIC_USER""Elasticsearch usernameELASTIC_PASSWORD""Elasticsearch passwordELASTIC_INDEX""Elasticsearch index pattern to search documents inELASTIC_BATCH_SIZE3000How much documents to extract from ES in one batchELASTIC_TIMEOUT120Elasticsearchsearchquery timeoutELASTIC_MAX_DATEUpper date limit (format is the same as @timestamp field)ELASTIC_TIMESTAMP_FIELD@timestampName of timesteamp field in ElasticsearchLOKI_URLhttp://localhost:3100Loki instance URLLOKI_USERNAME""Loki usernameLOKI_PASSWORD""Loki passwordLOKI_TENANT_ID""Loki Tenant ID (Org ID)LOKI_BATCH_SIZE1048576Maximum batch size (in bytes)LOKI_POOL_LOAD_FACTOR10Maximum number of push non-waiting requestsLOKI_PUSH_MODEpbpb- protobuf + snappy,gzip- json + gzip,json- just jsonLOKI_WAIT_TIMEOUT0How much time (in seconds) to wait after a Loki push requestSTATE_MODEnoneConfigures es2loki persistence (dbis recommended). Usenoneto disable persistence completelySTATE_START_OVERClean up persisted data and start overSTATE_DB_URLpostgres://127.0.0.1:5432/postgresDatabase URL fordbpersistence