Project: Loggie (Observability and Analysis / Observability)
File: CHANGELOG-v1.md

. If upgrading from an earlier version to v1.5, be sure to check whether `db` has been configured for `file source`. If it is not configured, you can just ignore it, and the default value will remain compatible. Added rocketmq sink (#530) Added franzKafka source (#573) Added kata runtime (#554) `typePodFields`/`typeNodeFields` is supported in LogConfig/ClusterLogConfig (#450) sink codec support printEvents (#448) Added queue in LogConfig/ClusterLogConfig (#457) Changed `olivere/elastic` to the official elasticsearch go client (#581) Supported `copytruncate` in file source (#571) Added `genfiles` sub command (#471) Added queue in LogConfig/ClusterLogConfig queue (#457) Added `sortBy` field in elasticsearch source (#473) Added host VM mode with Kubernetes as the configuration center (#449) (#489) New `addHostMeta` interceptor (#474) Added persistence driver `badger` (#475) (#584) Ignore LogConfig with sidecar injection annotation (#478) Added `toStr` action in transformer interceptor (#482) You can mount the root directory of a node to the Loggie container without mounting additional Loggie volumes (#460) Get loggie version with api and sub command (#496) (#508) Added the `worker` and the `clientId` in Kafka source (#506) (#507) Upgrade `kafka-go` version (#506) (#567) Added resultStatus in dev sink which can be used to simulate failure, drop (#531) Pretty error when unmarshal yaml configuration failed (#539) Added default topic if render kafka topic failed (#550) Added `ignoreUnknownTopicOrPartition` in kafka sink (#560) Supported multiple topics in kafka source (#548) Added default index if render elasticsearch index failed (#551) (#553) The default `maxOpenFds` is set to 4096 (#559) Supported default `sinkRef` in kubernetes discovery (#555) Added `${_k8s.clusterlogconfig}` in `typePodFields` (#569) Supported omit empty fields in Kubernetes discovery (#570) Optimizes `maxbytes` interceptors (#575) Moved `readFromTail`, `cleanFiles`, `fdHoldTimeoutWhenInactive`, `fdHoldTimeoutWhenRemove` from watcher to outer layer in `file source` (#579) (#585) Added `cheanUnfinished` in cleanFiles (#580) Added `target` in `maxbyte` interceptor (#588) Added `partionKey` in franzKafka (#562) Added `highPrecision` in `rateLimit` interceptor (#525) Fixed panic when kubeEvent Series is nil (#459) Upgraded `automaxprocs` version to v1.5.1 (#488) Fixed set defaults failed in `fieldsUnderKey` (#513) Fixed parse condition failed when contain ERROR in transformer interceptor (#514) (#515) Fixed grpc batch out-of-order data streams (#517) Fixed large line may cause oom (#529) Fixed duplicated batchSize in queue (#533) Fixed sqlite locked panic (#524) Fixed command can't be used in multi-arch container (#541) Fixed `logger listener` may cause block (#561) (#552) Fixed `sink concurrency` deepCopy failed (#563) Drop events when partial error in elasticsearch sink (#572) Added Loggie dashboard feature for easier troubleshooting (#416) Enhanced log alerting function with more flexible log alert detection rules and added alertWebhook sink (#392) Added sink concurrency support for automatic adaptation based on downstream delay (#376) Added franzKafka sink for users who prefer the franz kafka library (#423) Added elasticsearch source (#345) Added zinc sink (#254) Added pulsar sink (#417) Added grok action to transformer interceptor (#418) Added split action to transformer interceptor (#411) Added jsonEncode action to transformer interceptor (#421) Added fieldsFromPath configuration to source for obtaining fields from file content (#401) Added fieldsRef parameter to filesource listener for obtaining key value from fields configuration and adding to metrics as label (#402) In transformer interceptor, added dropIfError support to drop event if action execution

(#409) Added info listener which currently exposes loggieinfostat metrics and displays version label (#410) Added support for customized kafka sink partition key Added sasl support to Kafka source (#415) Added https insecureSkipVerify support to loki sink (#422) Optimized file source for large files (#430) Changed default value of file source maxOpenFds to 1024 (#437) ContainerRuntime can now be set to none (#439) Upgraded to go 1.18 (#440) Optimize the configuration parameters to remove the redundancy generated by rendering Added source fields to filesource listener (#402) Fixed issue of transformer copy action not copying non-string body (#420) Added fetching of logs file from UpperDir when rootfs collection is enabled (#414) Fix pipeline restart npe (#454) Fix create dir soft link job (#453) Added Loggie dashboard feature for easier troubleshooting (#416) Enhanced log alerting function with more flexible log alert detection rules and added alertWebhook sink (#392) Added sink concurrency support for automatic adaptation based on downstream delay (#376) Added franzKafka sink for users who prefer the franz kafka library (#423) Added elasticsearch source (#345) Added zinc sink (#254) Added pulsar sink (#417) Added grok action to transformer interceptor (#418) Added split action to transformer interceptor (#411) Added jsonEncode action to transformer interceptor (#421) Added fieldsFromPath configuration to source for obtaining fields from file content (#401) Added fieldsRef parameter to filesource listener for obtaining key value from fields configuration and adding to metrics as label (#402) In transformer interceptor, added dropIfError support to drop event if action execution fails (#409) Added info listener which currently exposes loggieinfostat metrics and displays version label (#410) Added support for customized kafka sink partition key Added sasl support to Kafka source (#415) Added https insecureSkipVerify support to loki sink (#422) Optimized file source for large files (#430) Changed default value of file source maxOpenFds to 1024 (#437) ContainerRuntime can now be set to none (#439) Upgraded to go 1.18 (#440) Added source fields to filesource listener (#402) Fixed issue of transformer copy action not copying non-string body (#420) Added fetching of logs file from UpperDir when rootfs collection is enabled (#414) Add transformer interceptor (#267) Add api /api/v1/helper for debugging (#291) Monitoring listener for normalize interceptor (#224) Add typeNodeFields in kubernetes discovery, change k8sFields to typePodFields, add pod.uid, pod.container.id, pod.container.image to typePodFields (#348) File source support any lineDelimiter (#261) Logconfig/Clusterlogconfig labelSelector support wildcard * (#269) Add override annotation in logConfig/clusterLogConfig (#269) Add go test in makefile and github action (#282) Support total events in dev source (#307) Make file sink default file permissions from 600 to 644 (#304) Change goccy/go-yaml to gopkg.in/yaml.v2 (#317) Add schema interceptor (#340) Support collect log files from pod pv (#341) Support dynamic container logs (#334) Put the encoding conversion module in the source (#344) Support [a.b] when fields key is a.b other than a nested fields #301 #297 Loki sink will replace fields including string '. / -' to '_' (#347) Set beatsFormat timestamp to UTC in sink codec (#268) Fix pipeline stop block (#280)(#284) (#289) (#305)(#309) (#335) Fix addonMeta panic (#285) Move private fields from header to meta in grpc source (#286) Fix clusterlogconfig npe when deleting (#306) Fix prometheus export topic (#314) Fix watchTask stop panic; fix big line collect

(#315) Fix type assert panic when deleting pod recieve DeletedFinalStateUnknown objects (#323) Fix maxbytes order (#326) Check if pod is ready when handle logconfig events (#328) Stop pipeline when config is empty (#330) Change leaderElection Identity in kubeEvent source (#349) Fix ignoreOlder missing in file source config Flush clusterlogconfig configuration when sink/interceptor is being updated Fix the problem of missing stream and time when collecting stdout logs (#363) Fix kubeEvent source when lost leader connection (#361) Do not ignore pod events when container id is changed (#377) Validate multiline pattern (#381) LabelSelector in LogConfig can be updated now (#369) Fix interceptor order in pipelines (#387) Add transformer interceptor (#267) Add api /api/v1/helper for debugging (#291) Monitoring listener for normalize interceptor (#224) Add typeNodeFields in kubernetes discovery, change k8sFields to typePodFields, add pod.uid, pod.container.id, pod.container.image to typePodFields (#348) File source support any lineDelimiter (#261) Logconfig/Clusterlogconfig labelSelector support wildcard * (#269) Add override annotation in logConfig/clusterLogConfig (#269) Add go test in makefile and github action (#282) Support total events in dev source (#307) Make file sink default file permissions from 600 to 644 (#304) Change goccy/go-yaml to gopkg.in/yaml.v2 (#317) Add schema interceptor (#340) Support collect log files from pod pv (#341) Support dynamic container logs (#334) Put the encoding conversion module in the source (#344) Support [a.b] when fields key is a.b other than a nested fields #301 #297 Loki sink will replace fields including string '. / -' to '_' (#347) Set beatsFormat timestamp to UTC in sink codec (#268) Fix pipeline stop block (#280)(#284) (#289) (#305)(#309) (#335) Fix addonMeta panic (#285) Move private fields from header to meta in grpc source (#286) Fix clusterlogconfig npe when deleting (#306) Fix prometheus export topic (#314) Fix watchTask stop panic; fix big line collect block (#315) Fix type assert panic when deleting pod recieve DeletedFinalStateUnknown objects (#323) Fix maxbytes order (#326) Check if pod is ready when handle logconfig events (#328) Stop pipeline when config is empty (#330) Change leaderElection Identity in kubeEvent source (#349) Support container root filesystem log collection, so Loggie could collect the log files in container even if the pod is not mounting any volumes. #209 Add `sls` sink for aliyun. #172 Support specifying more time intervals(like 1d, 1w, 1y) for the "ignoreOlder" config of file source. #153 (by @iamyeka) Add `ignoreError` in normalize interceptor. #205 Add system listener, which can export CPU/Memory metrics of Loggie. #88 Set `log.jsonFormat` default to false, so Loggie would no longer print json format logs in default. #186 Add `callerSkipCount` and `noColor` logger flag. #185 `file sink` support codec. #215 (by @lyp256) Support linux arm64. #229 (by @ziyu-zhao) `addk8smeta` support adding workload metadata. #221 (by @lyp256) Support extra labels in prometheusExporter source. #233 Logconfig matchFields labels, annotations and env support wildcards(*). #235 (by @LeiZhang-Hunter) Add source codec for file source. #239 Some specific fields pattern now supports `${env.XX}`, `${+date}`, `${object}`, `${k8s.XX}`. #248 We support rename `body` of events in normalize interceptor now. #238 Use local time format in pattern fields. #234 Refactor reader process chain. #166 Optimize merge strategy of default interceptors. #174 (by @iamyeka) Merge defaults configurations support recursion. #212 Cache env for optimizing performance when using fieldsFromEnv. #178 (by @iamyeka) Update yaml lib to

#242 Oomit empty pipeline configurations. #244 Fix json error with go 1.18. #164 Fix list logconfigs by namespace when handling pod events. #173 Fix watcher stop timeout in file source. #188 Fix MultiFileWriter in file sink. #202 (by @lyp256) Fix pipeline stop may block. #207 Fix unstable interceptor sort. #216 Fix grpc source ack panic. #218 Fix sub path in kubernetes discovery. #236 (by @yuzhichang) Loki sink supported, so you can send messages to Loki now. #119 > eg: > > ```yaml > sink: > type: loki > url: "http://localhost:3100/loki/api/v1/push" > ``` Add spec.interceptors and spec.sink fields in ClusterLogConfig/LogConfig CRD #120 > eg: > > ```yaml > apiVersion: loggie.io/v1beta1 > kind: LogConfig > metadata: > name: tomcat > namespace: default > spec: > selector: > type: pod > labelSelector: > app: tomcat > pipeline: > sources: | > - type: file > name: common > paths: > - stdout > sink: | > type: dev > printEvents: false > interceptors: | > - type: rateLimit > qps: 90000 > ``` A new codec type `raw` is added. #137 > eg: > > ```yaml > sink: > type: dev > codec: > type: raw > ``` A new approach to external configurations, `-config.from=env`, allows Loggie to read configs from env. #132 (by @chaunceyjiang) Normalize interceptor now support `fmt` processor. This can reformat the fields by patterns. #141 > eg: > > ```yaml > interceptors: > - type: normalize > processors: > - fmt: > fields: > d: new-${a.b}-${c} > ``` Adding `enabled` fields in source/interceptor. #140 We would flush the pipelines.yml immediately when updating sink/interceptor CR. #139 Upgraded client-go version to 0.22.2 #99 (by @fengshunli) `sniff` are disabled in elasticsearch sink by default. #138 Updated kubeEvent source #144 leader selection support. blacklist of namespaces support. adding `watchLatestEvents` fields, allows Loggie watch the lastest events from K8s. > eg: > > ```yaml > - type: kubeEvent > name: eventer > kubeconfig: /.kube/config > watchLatestEvents: true > blackListNamespaces: ["default"] > ``` Brace Expansion{alt1,...} suppport, matches a sequence of characters if one of the comma-separated alternatives matches. > eg: > > log files following: > > 1. /tmp/loggie/service/order/access.log > 2. /tmp/loggie/service/order/access.log.2022-04-11 > 3. /tmp/loggie/service/pay/access.log > 4. /tmp/loggie/service/pay/access.log.2022-04-11 > > then the path field in filesource could be: /tmp/loggie//access.log{,.--} Adding `local` in normalize interceptor, in case we want to parse timestamp with local time. We support a simple way to add file state when collecting log files, enable `addonMeta`, then Loggie will add more metadata of files in events. > eg: > > ```json > { > "body": "this is test", > "state": { > "pipeline": "local", > "source": "demo", > "filename": "/var/log/a.log", > "timestamp": "2006-01-02T15:04:05.000Z", > "offset": 1024, > "bytes": 4096, > "hostname": "node-1" > } > } > ``` Adding `pod.ip``node.ip` in K8s discovery. Customized `documentId` supported in elasticsearch sink. Fix read chan size error in filesource #102 Update default value of `maxAcceptedBytes` in kafka source which may cause panic in component starting. #112 First Loggie Release! :sparkler: add components: sources: file kafka grpc unix kubeEvent prometheusExporter dev interceptors: normalize rateLimit addK8sMeta logAlert metric retry maxbytes sink: elasticsearch kafka grpc file dev discovery: Kubernetes discovery, introduced CRD: ClusterLogConfig/LogConfig/Interceptor/Sink monitor: filesource filewatcher logAlert queue sink reload