Repository: banzaicloud/logging-operator
Pull request #235 (2019-11-14 23:19:00)

Exception detector plugin
| Q               | A
| --------------- | ---
| Bug fix?        | no
| New feature?    | yes
| API breaks?     | no
| Deprecations?   | no
| Related tickets | fixes #236 
| License         | Apache 2.0


### Checklist
- [x] Error handling code meets the [guideline](https://github.com/banzaicloud/pipeline/blob/master/docs/error-handling-guide.md)
- [x] Logging code meets the guideline (TODO)
- [x] User guide and development docs updated (if needed)
- [ ] Related Helm chart(s) updated (if needed)




Patch:
diff --git a/.idea/copyright/Banzai_Cloud.xml b/.idea/copyright/Banzai_Cloud.xml
index 676bad7c..f3620400 100644
--- a/.idea/copyright/Banzai_Cloud.xml
+++ b/.idea/copyright/Banzai_Cloud.xml
@@ -1,6 +1,6 @@
 <component name="CopyrightManager">
-    <copyright>
-        <option name="notice" value="Copyright © &amp;#36;today.year Banzai Cloud&#10;&#10;Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);&#10;you may not use this file except in compliance with the License.&#10;You may obtain a copy of the License at&#10;&#10;    http://www.apache.org/licenses/LICENSE-2.0&#10;&#10;Unless required by applicable law or agreed to in writing, software&#10;distributed under the License is distributed on an &quot;AS IS&quot; BASIS,&#10;WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.&#10;See the License for the specific language governing permissions and&#10;limitations under the License." />
-        <option name="myName" value="Banzai Cloud" />
-    </copyright>
+  <copyright>
+    <option name="notice" value="Copyright © &amp;#36;today.year Banzai Cloud&#10;&#10;Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);&#10;you may not use this file except in compliance with the License.&#10;You may obtain a copy of the License at&#10;&#10;   http://www.apache.org/licenses/LICENSE-2.0&#10;&#10;Unless required by applicable law or agreed to in writing, software&#10;distributed under the License is distributed on an &quot;AS IS&quot; BASIS,&#10;WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.&#10;See the License for the specific language governing permissions and&#10;limitations under the License." />
+    <option name="myName" value="Banzai Cloud" />
+  </copyright>
 </component>
\ No newline at end of file
diff --git a/README.md b/README.md
index b79b83d1..28782d75 100644
--- a/README.md
+++ b/README.md
@@ -119,8 +119,9 @@ For complete list of supported plugins please check the [plugins index](/docs/pl
 | [Sumologic](./docs/plugins/outputs/sumologic.md)          | Output | Send your logs to Sumologic                            |    GA   | [1.6.1](https://github.com/SumoLogic/fluentd-output-sumologic/releases/tag/1.6.1)       |
 | [CloudWatch](./docs/plugins/outputs/cloudwatch.md)        | Output | Send your logs to AWS CloudWatch                       |    GA   | [0.7.4](https://github.com/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs/releases/tag/v0.7.4) |
 | [Kafka](./docs/plugins/outputs/kafka.md)                  | Output | Send your logs to Kafka                                |    GA   | [0.12.1](https://github.com/fluent/fluent-plugin-kafka/releases/tag/v0.12.1)            |
-| [Tag Normaliser](./docs/plugins/filters/tagnormaliser.md) | Parser | Normalise tags for outputs                             |    GA   |                                                                                         |
-| [Parser](./docs/plugins/filters/parser.md)                | Parser | Parse logs with parser plugin                          |    GA   |                                                                                         |
+| [Exception Detector](./docs/plugins/outputs/detect_exceptions.md)| Filter | Exception detector plugin for fluentd           |    GA   | [0.0.13](https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions/releases/tag/0.0.13) |
+| [Tag Normaliser](./docs/plugins/filters/tagnormaliser.md) | Filter | Normalise tags for outputs                             |    GA   |                                                                                         |
+| [Parser](./docs/plugins/filters/parser.md)                | Filter | Parse logs with parser plugin                          |    GA   |                                                                                         |
 
 ---
 
diff --git a/charts/logging-operator-logging/README.md b/charts/logging-operator-logging/README.md
index 9ea224e8..9ae58dda 100644
--- a/charts/logging-operator-logging/README.md
+++ b/charts/logging-operator-logging/README.md
@@ -25,7 +25,7 @@ The following tables lists the configurable parameters of the logging-operator-l
 | `fluentbit.image.repository`                        | Fluentbit container image repository                   | `fluent/fluent-bit`                                        |
 | `fluentbit.image.pullPolicy`                        | Fluentbit container pull policy                        | `IfNotPresent`                                             |
 | `fluentd.enabled`                                   | Install fluentd                                        | true                                                       |
-| `fluentd.image.tag`                                 | Fluentd container image tag                            | `v1.7.4-alpine-6`                                          |
+| `fluentd.image.tag`                                 | Fluentd container image tag                            | `v1.7.4-alpine-7`                                          |
 | `fluentd.image.repository`                          | Fluentd container image repository                     | `banzaicloud/fluentd`                                      |
 | `fluentd.image.pullPolicy`                          | Fluentd container pull policy                          | `IfNotPresent`                                             |
 | `fluentd.volumeModImage.tag`                        | Fluentd volumeModImage container image tag             | `latest`                                                   |
diff --git a/charts/logging-operator/README.md b/charts/logging-operator/README.md
index 7c0d7f8f..f94cd001 100644
--- a/charts/logging-operator/README.md
+++ b/charts/logging-operator/README.md
@@ -106,7 +106,7 @@ The following tables lists the configurable parameters of the logging-operator-l
 | `fluentbit.image.repository`                        | Fluentbit container image repository                   | `fluent/fluent-bit`            |
 | `fluentbit.image.pullPolicy`                        | Fluentbit container pull policy                        | `IfNotPresent`                 |
 | `fluentd.enabled`                                   | Install fluentd                                        | true                           |
-| `fluentd.image.tag`                                 | Fluentd container image tag                            | `v1.7.4-alpine-6`              |
+| `fluentd.image.tag`                                 | Fluentd container image tag                            | `v1.7.4-alpine-7`              |
 | `fluentd.image.repository`                          | Fluentd container image repository                     | `banzaicloud/fluentd`          |
 | `fluentd.image.pullPolicy`                          | Fluentd container pull policy                          | `IfNotPresent`                 |
 | `fluentd.volumeModImage.tag`                        | Fluentd volumeModImage container image tag             | `latest`                       |
diff --git a/config/crd/bases/logging.banzaicloud.io_clusterflows.yaml b/config/crd/bases/logging.banzaicloud.io_clusterflows.yaml
index aa1af843..61966727 100644
--- a/config/crd/bases/logging.banzaicloud.io_clusterflows.yaml
+++ b/config/crd/bases/logging.banzaicloud.io_clusterflows.yaml
@@ -108,6 +108,39 @@ spec:
                         description: Separator (default:_)
                         type: string
                     type: object
+                  detectExceptions:
+                    properties:
+                      languages:
+                        description: 'Programming languages for which to detect exceptions.
+                          (default: [])'
+                        items:
+                          type: string
+                        type: array
+                      max_bytes:
+                        description: 'Maximum number of bytes to flush (0 means no
+                          limit) (default: 0)'
+                        type: integer
+                      max_lines:
+                        description: 'Maximum number of lines to flush (0 means no
+                          limit) (default: 1000)'
+                        type: integer
+                      message:
+                        description: 'The field which contains the raw message text
+                          in the input JSON data. (default: "")'
+                        type: string
+                      multiline_flush_interval:
+                        description: 'The interval of flushing the buffer for multiline
+                          format. (default: nil)'
+                        type: string
+                      remove_tag_prefix:
+                        description: 'The prefix to be removed from the input tag
+                          when outputting a record. (default: "")'
+                        type: string
+                      stream:
+                        description: 'Separate log streams by this field in the input
+                          JSON data. (default: "")'
+                        type: string
+                    type: object
                   geoip:
                     properties:
                       backend_library:
diff --git a/config/crd/bases/logging.banzaicloud.io_flows.yaml b/config/crd/bases/logging.banzaicloud.io_flows.yaml
index 99b5d60d..06996cd2 100644
--- a/config/crd/bases/logging.banzaicloud.io_flows.yaml
+++ b/config/crd/bases/logging.banzaicloud.io_flows.yaml
@@ -113,6 +113,39 @@ spec:
                         description: Separator (default:_)
                         type: string
                     type: object
+                  detectExceptions:
+                    properties:
+                      languages:
+                        description: 'Programming languages for which to detect exceptions.
+                          (default: [])'
+                        items:
+                          type: string
+                        type: array
+                      max_bytes:
+                        description: 'Maximum number of bytes to flush (0 means no
+                          limit) (default: 0)'
+                        type: integer
+                      max_lines:
+                        description: 'Maximum number of lines to flush (0 means no
+                          limit) (default: 1000)'
+                        type: integer
+                      message:
+                        description: 'The field which contains the raw message text
+                          in the input JSON data. (default: "")'
+                        type: string
+                      multiline_flush_interval:
+                        description: 'The interval of flushing the buffer for multiline
+                          format. (default: nil)'
+                        type: string
+                      remove_tag_prefix:
+                        description: 'The prefix to be removed from the input tag
+                          when outputting a record. (default: "")'
+                        type: string
+                      stream:
+                        description: 'Separate log streams by this field in the input
+                          JSON data. (default: "")'
+                        type: string
+                    type: object
                   geoip:
                     properties:
                       backend_library:
diff --git a/docs/crds.md b/docs/crds.md
index 4ac22151..54da71b5 100644
--- a/docs/crds.md
+++ b/docs/crds.md
@@ -169,7 +169,7 @@ spec:
   fluentd: 
     image:
       repository: banzaicloud/fluentd
-      tag: v1.7.4-alpine-6
+      tag: v1.7.4-alpine-7
       pullPolicy: IfNotPresent
   fluentbit: {}
   controlNamespace: logging
diff --git a/docs/plugins/Readme.md b/docs/plugins/Readme.md
index 0ab611de..e31c635e 100644
--- a/docs/plugins/Readme.md
+++ b/docs/plugins/Readme.md
@@ -5,6 +5,7 @@
 
 - [concat](./filters/concat.md)
 - [dedot](./filters/dedot.md)
+- [detect_exceptions](./filters/detect_exceptions.md)
 - [geoip](./filters/geoip.md)
 - [parser](./filters/parser.md)
 - [record_transformer](./filters/record_transformer.md)
diff --git a/docs/plugins/filters/detect_exceptions.md b/docs/plugins/filters/detect_exceptions.md
new file mode 100644
index 00000000..fbb07758
--- /dev/null
+++ b/docs/plugins/filters/detect_exceptions.md
@@ -0,0 +1,27 @@
+# Exception Detector
+## Overview
+This filter plugin consumes a log stream of JSON objects which contain single-line log messages. If a consecutive sequence of log messages form an exception stack trace, they forwarded as a single, combined JSON object. Otherwise, the input log data is forwarded as is.
+More info at https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions
+
+ > Note: As Tag management is not supported yet, this Plugin is **mutually exclusive** with [Tag normaliser](./tagnormaliser.md)
+
+ #### Example output configurations
+ ```
+filters:
+  spec:
+    detectExceptions:
+      languages: java, python
+      multiline_flush_interval: 0.1
+ ```
+
+## Configuration
+### Output Config
+| Variable Name | Type | Required | Default | Description |
+|---|---|---|---|---|
+| message | string | No |  "" | The field which contains the raw message text in the input JSON data. <br> |
+| remove_tag_prefix | string | No |  "" | The prefix to be removed from the input tag when outputting a record. <br> |
+| multiline_flush_interval | string | No |  nil | The interval of flushing the buffer for multiline format. <br> |
+| languages | []string | No |  [] | Programming languages for which to detect exceptions. <br> |
+| max_lines | int | No |  1000 | Maximum number of lines to flush (0 means no limit) <br> |
+| max_bytes | int | No |  0 | Maximum number of bytes to flush (0 means no limit) <br> |
+| stream | string | No |  "" | Separate log streams by this field in the input JSON data. <br> |
diff --git a/pkg/sdk/api/v1beta1/flow_types.go b/pkg/sdk/api/v1beta1/flow_types.go
index 0c13be43..f0d645b5 100644
--- a/pkg/sdk/api/v1beta1/flow_types.go
+++ b/pkg/sdk/api/v1beta1/flow_types.go
@@ -36,6 +36,7 @@ type Filter struct {
 	RecordTransformer *filter.RecordTransformer  `json:"record_transformer,omitempty"`
 	GeoIP             *filter.GeoIP              `json:"geoip,omitempty"`
 	Concat            *filter.Concat             `json:"concat,omitempty"`
+	DetectExceptions  *filter.DetectExceptions   `json:"detectExceptions,omitempty"`
 }
 
 // FlowStatus defines the observed state of Flow
diff --git a/pkg/sdk/api/v1beta1/logging_types.go b/pkg/sdk/api/v1beta1/logging_types.go
index 8c618bb3..26de8a7a 100644
--- a/pkg/sdk/api/v1beta1/logging_types.go
+++ b/pkg/sdk/api/v1beta1/logging_types.go
@@ -78,7 +78,7 @@ func (l *Logging) SetDefaults() (*Logging, error) {
 			copy.Spec.FluentdSpec.Image.Repository = "banzaicloud/fluentd"
 		}
 		if copy.Spec.FluentdSpec.Image.Tag == "" {
-			copy.Spec.FluentdSpec.Image.Tag = "v1.7.4-alpine-6"
+			copy.Spec.FluentdSpec.Image.Tag = "v1.7.4-alpine-7"
 		}
 		if copy.Spec.FluentdSpec.Image.PullPolicy == "" {
 			copy.Spec.FluentdSpec.Image.PullPolicy = "IfNotPresent"
diff --git a/pkg/sdk/api/v1beta1/zz_generated.deepcopy.go b/pkg/sdk/api/v1beta1/zz_generated.deepcopy.go
index 608b9f1c..ef7f0574 100644
--- a/pkg/sdk/api/v1beta1/zz_generated.deepcopy.go
+++ b/pkg/sdk/api/v1beta1/zz_generated.deepcopy.go
@@ -217,6 +217,11 @@ func (in *Filter) DeepCopyInto(out *Filter) {
 		*out = new(filter.Concat)
 		**out = **in
 	}
+	if in.DetectExceptions != nil {
+		in, out := &in.DetectExceptions, &out.DetectExceptions
+		*out = new(filter.DetectExceptions)
+		(*in).DeepCopyInto(*out)
+	}
 }
 
 // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Filter.
diff --git a/pkg/sdk/model/filter/detect_exceptions.go b/pkg/sdk/model/filter/detect_exceptions.go
new file mode 100644
index 00000000..5081fe7c
--- /dev/null
+++ b/pkg/sdk/model/filter/detect_exceptions.go
@@ -0,0 +1,76 @@
+// Copyright © 2019 Banzai Cloud
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package filter
+
+import (
+	"github.com/banzaicloud/logging-operator/pkg/sdk/model/secret"
+	"github.com/banzaicloud/logging-operator/pkg/sdk/model/types"
+)
+
+// +docName:"Exception Detector"
+//This filter plugin consumes a log stream of JSON objects which contain single-line log messages. If a consecutive sequence of log messages form an exception stack trace, they forwarded as a single, combined JSON object. Otherwise, the input log data is forwarded as is.
+//More info at https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions
+//
+// > Note: As Tag management is not supported yet, this Plugin is **mutually exclusive** with [Tag normaliser](./tagnormaliser.md)
+//
+// #### Example output configurations
+// ```
+//filters:
+//  spec:
+//    detectExceptions:
+//      languages: java, python
+//      multiline_flush_interval: 0.1
+// ```
+type _docExceptionDetector interface{}
+
+// +kubebuilder:object:generate=true
+// +docName:"Output Config"
+type DetectExceptions struct {
+	// The field which contains the raw message text in the input JSON data. (default: "")
+	Message string `json:"message,omitempty"`
+	// The prefix to be removed from the input tag when outputting a record. (default: "")
+	RemoveTagPrefix string `json:"remove_tag_prefix,omitempty"`
+	// The interval of flushing the buffer for multiline format. (default: nil)
+	MultilineFlushInterval string `json:"multiline_flush_interval,omitempty"`
+	// Programming languages for which to detect exceptions. (default: [])
+	Languages []string `json:"languages,omitempty"`
+	// Maximum number of lines to flush (0 means no limit) (default: 1000)
+	MaxLines int `json:"max_lines,omitempty"`
+	// Maximum number of bytes to flush (0 means no limit) (default: 0)
+	MaxBytes int `json:"max_bytes,omitempty"`
+	// Separate log streams by this field in the input JSON data. (default: "")
+	Stream string `json:"stream,omitempty"`
+}
+
+func (d *DetectExceptions) ToDirective(secretLoader secret.SecretLoader, id string) (types.Directive, error) {
+	pluginType := "detect_exceptions"
+	pluginID := id + "_" + pluginType
+	detector := &types.OutputPlugin{
+		PluginMeta: types.PluginMeta{
+			Type:      pluginType,
+			Directive: "match",
+			Tag:       "kubernetes.**",
+			Id:        pluginID,
+		},
+	}
+	detect := d.DeepCopy()
+	detect.RemoveTagPrefix = "kubernetes"
+	if params, err := types.NewStructToStringMapper(secretLoader).StringsMap(detect); err != nil {
+		return nil, err
+	} else {
+		detector.Params = params
+	}
+	return detector, nil
+}
diff --git a/pkg/sdk/model/filter/zz_generated.deepcopy.go b/pkg/sdk/model/filter/zz_generated.deepcopy.go
index 9aa69a6a..fa8b492f 100644
--- a/pkg/sdk/model/filter/zz_generated.deepcopy.go
+++ b/pkg/sdk/model/filter/zz_generated.deepcopy.go
@@ -50,6 +50,26 @@ func (in *DedotFilterConfig) DeepCopy() *DedotFilterConfig {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *DetectExceptions) DeepCopyInto(out *DetectExceptions) {
+	*out = *in
+	if in.Languages != nil {
+		in, out := &in.Languages, &out.Languages
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new DetectExceptions.
+func (in *DetectExceptions) DeepCopy() *DetectExceptions {
+	if in == nil {
+		return nil
+	}
+	out := new(DetectExceptions)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *GeoIP) DeepCopyInto(out *GeoIP) {
 	*out = *in


Test patch:
diff --git a/pkg/sdk/model/filter/detect_exceptions_test.go b/pkg/sdk/model/filter/detect_exceptions_test.go
new file mode 100644
index 00000000..6a39ed42
--- /dev/null
+++ b/pkg/sdk/model/filter/detect_exceptions_test.go
@@ -0,0 +1,45 @@
+// Copyright © 2019 Banzai Cloud
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package filter_test
+
+import (
+	"testing"
+
+	"github.com/banzaicloud/logging-operator/pkg/sdk/model/filter"
+	"github.com/banzaicloud/logging-operator/pkg/sdk/model/render"
+	"github.com/ghodss/yaml"
+)
+
+func TestDetectExceptions(t *testing.T) {
+	CONFIG := []byte(`
+multiline_flush_interval: 0.1
+languages: 
+  - java
+  - python
+`)
+	expected := `
+<match kubernetes.**>
+  @type detect_exceptions
+  @id test_detect_exceptions
+  languages ["java","python"]
+  multiline_flush_interval 0.1
+  remove_tag_prefix kubernetes
+</match>
+`
+	ed := &filter.DetectExceptions{}
+	yaml.Unmarshal(CONFIG, ed)
+	test := render.NewOutputPluginTest(t, ed)
+	test.DiffResult(expected)
+}