Change:
In CPU usage went up significantly compared to It consumes more than one core even when there's no significant logging load on the node. Two changes between those version: CC
Ruby by default is single-core, so anything over that is likely introduced by the changes. But we'd need to investigate to make sure.
Can you please disable monitoring and see whether this is the problem?
I meant can you ...
Was going to do this, yes
Built container manually without prometheus plugin, still consumes 1.7 cpu
Can you please find the difference then?
Used new image with the old configuration (), still 1.7 cores It seems like a problem with the new image for some reason. FYI
What can consume 1.7 cores with the same configuration on a different machine? How to debug this?
Does this mean all components are same, ruby version, plugins and configuration settings? What are the differences between old and new image? OS or other parts are also different?
There are at least the following changer Ubuntu Trusty -Debian Jessie td-agent package -manually installed embedded ruby -ruby 2.1 embedded gems -fixed version gems from Gemfile embedded libjemalloc - I understand that it's a lot of changes and I'm going to narrow the problem down, the main question is where to look and how to debug to speed this process up?
If this is not fixed before Monday we will revert . cc
Only and should be reverted, not
No, we want to revert migration to the new base image. The changes to extract configuration into env var/config map should preserve. This mean that the revert might not be straightforward.
I think our backup option is to switch back to using td-agent package, not reverting whole change, because is also about security and we don't want to revert that I will check whether that helps
Thanks for investigating this! I have a couple comments: The full package diff can be found here: The first thing I would look at is whether the ruby version makes a difference. I believe td-agent was using 2.3 The main benefit of is actually not using td-agent, since the td-agent package includes lots of stuff we don't use (e.g. rails). Maybe if we need to fallback we could manually remove those extra packages. I'll poke around in the container and see if anything jumps out.
Thanks for the comments, they totally make sense. Also, thanks for the diff. Thanks!
Actually td-agent was using ruby 2.1:
A manual search through commenting out parts of the config file revealed that removing the systemd config solves the issue: I'll play around with the systemd libraries and see if I can figure out what's going on here.
Still investigating, but I think this may be a regression in the fluent-plugin-systemd library between version 0.0.5 (old image) and 0.0.7 (new image). I realized the diff I linked above was not in sync with the version that was actually pushed (). That diff is here, and accurately reflects the change in the systemd library:
Confirmed this is a regression in (0.0.6 does not have the problem). Fortunately 0.0.7 only has a single commit, I'll file an issue on the plugin. In the meantime, I'll send a PR to pin fluent-plugin-systemd to 0.0.6.
Thanks a huge lot for investigating and fixing this!
Reopening to track the version bump.
Good investigation. Latest systemd plugin uses and it seems to cause busy loop like problem.
Hi I just released that should fix this issue.
Thanks! And thanks to who already included it into our image :)
Yeah thanks your too fast for me :)
Thank you a lot for the investigation.

Discussion:
policies.  An API call must be allowed by some policy in order to proceed. The following implementations are available, and are selected by flag: <del> - `--authoriation_mode=AlwaysDeny` - `--authoriation_mode=AlwaysAllow` - `--authoriation_mode=ABAC` </del> <ins> - `--authorization_mode=AlwaysDeny` - `--authorization_mode=AlwaysAllow` - `--authorization_mode=ABAC` </ins> `AlwaysDeny` blocks all requests (used in tests). `AlwaysAllow` allows all requests; use if you don't need authorization.