opened
lightkz
username_0
Based on discussion opening enhancement feature request.
https://discuss.elastic.co/t/reload-if-modification-time-changes-in-config-files/55556
Live reload config files without restarting filebeat process
issue
created
slava-vishnyakov
username_1
It would probably be better to reload, like nginx with something like `kill -s HUP`
comment
created
OferE
username_2
I'm working with microservices architecture and restarting the filebeat process is a big pain to me,
I have many services coming and going and i want them to register themselves with filebeats without restarting the filebeat process everytime a service joins.
Restart a process is a very aggressive operation and has consequences regarding the exactly once guarantees.
comment
created
ruflin
username_3
Does every service need its own config / prospector? Or do you know all of them in advance and could provide all the prospectors on first start?
comment
created
OferE
username_2
knowing all services in advance is a bad practice since i have multiple machine types, each type runs a different set of services. each machine may run different services even with the same machine type.
currently this is how i workaround everything: by providing one huge config.
The correct approach IMHO will be that every service will register itself with its own config - the current filebeat process will reload the configuration on every change/inroduction of new config and adjust in a clean way.
By the way - the outputs themselves should have the ability to change dynamically and accrding to each prospector, since some services may want to write to kafka (even into different topics) and others to elasctic/logstash.
This should be per prospector and not as a global definition.
The world is moving into microservices architectures - filebeat is a great project - but it is not a perfect match for micro services.
comment
created
ruflin
username_3
@username_2 I was also more thinking of current work arounds, but it seems you found one. It is definitively something we are looking into also related to https://github.com/elastic/beats/issues/464
comment
created
OferE
username_2
@username_3 - great to see that i am useful - also great to see that u hear community feedback! great product - i tend to use it in production for streaming messages to Kafka un huge scale.
comment
created
aqiao
username_4
hi guys, may i know the latest progress of this feature?
comment
created
ruflin
username_3
@username_4 Interesting timing. There is not directly a filebeat update but I just pushed this PR here: https://github.com/elastic/beats/pull/3281 This could be reused for filebeat prospectors in the future. It's a little bit trickier for filebeat as it must be ensured that a file is only harvested once and brings up the question when exactly a harvester should shut down ...
comment
created
ruflin
username_3
Closing as https://github.com/elastic/beats/pull/3362 was merged to master. This allows to dynamically reload prospectors.
comment
closed
ruflin
username_3
issue
created
OferE
username_2
@username_3 thanks for this feature!
comment