Project: VictoriaMetrics (Observability and Analysis / Observability)
File: Filebeat.md

weight: 1 title: Filebeat setup disableToc: true menu: docs: parent: "victorialogs-data-ingestion" weight: 1 aliases: /VictoriaLogs/data-ingestion/Filebeat.html Specify section in the `filebeat.yml` for sending the collected logs to : ```yaml output.elasticsearch: hosts: ["http://localhost:9428/insert/elasticsearch/"] parameters: msgfield: "message" timefield: "@timestamp" streamfields: "host.hostname,log.file.path" ``` Substitute the `localhost:9428` address inside `hosts` section with the real TCP address of VictoriaLogs. See for details on the `parameters` section. It is recommended verifying whether the initial setup generates the needed and uses the correct . This can be done by specifying `debug` and inspecting VictoriaLogs logs then: ```yaml output.elasticsearch: hosts: ["http://localhost:9428/insert/elasticsearch/"] parameters: msgfield: "message" timefield: "@timestamp" streamfields: "host.hostname,log.file.path" debug: "1" ``` If some must be skipped during data ingestion, then they can be put into `ignore_fields` . For example, the following config instructs VictoriaLogs to ignore `log.offset` and `event.original` fields in the ingested logs: ```yaml output.elasticsearch: hosts: ["http://localhost:9428/insert/elasticsearch/"] parameters: msgfield: "message" timefield: "@timestamp" streamfields: "host.name,log.file.path" ignore_fields: "log.offset,event.original" ``` When Filebeat ingests logs into VictoriaLogs at a high rate, then it may be needed to tune `worker` and `bulkmaxsize` options. For example, the following config is optimized for higher than usual ingestion rate: ```yaml output.elasticsearch: hosts: ["http://localhost:9428/insert/elasticsearch/"] parameters: msgfield: "message" timefield: "@timestamp" streamfields: "host.name,log.file.path" worker: 8 bulkmaxsize: 1000 ``` If the Filebeat sends logs to VictoriaLogs in another datacenter, then it may be useful enabling data compression via `compression_level` option. This usually allows saving network bandwidth and costs by up to 5 times: ```yaml output.elasticsearch: hosts: ["http://localhost:9428/insert/elasticsearch/"] parameters: msgfield: "message" timefield: "@timestamp" streamfields: "host.name,log.file.path" compression_level: 1 ``` By default, the ingested logs are stored in the `(AccountID=0, ProjectID=0)` . If you need storing logs in other tenant, then specify the needed tenant via `headers` at `output.elasticsearch` section. For example, the following `filebeat.yml` config instructs Filebeat to store the data to `(AccountID=12, ProjectID=34)` tenant: ```yaml output.elasticsearch: hosts: ["http://localhost:9428/insert/elasticsearch/"] headers: AccountID: 12 ProjectID: 34 parameters: msgfield: "message" timefield: "@timestamp" streamfields: "host.name,log.file.path" ``` Filebeat checks a version of ElasticSearch on startup and refuses to start sending logs if the version is not compatible. In order to bypass this check please add `allowolderversions: true` into `output.elasticsearch` section: ```yaml output.elasticsearch: hosts: [ "http://localhost:9428/insert/elasticsearch/" ] parameters: msgfield: "message" timefield: "@timestamp" streamfields: "host.name,log.file.path" allowolderversions: true ``` Alternatively, is also possible to change version which VictoriaLogs reports to Filebeat by using `-elasticsearch.version` command-line flag. See also: . . . .