Repository: kentik/ktranslate
Pull request #368 (2022-07-26 21:00:36)

Tweaks to influx
Messing with influx to change how attributes are set. 

```
kentik.SYNOLOGY-SYSTEM-MIB,device_ip=10.10.0.251,modelName=DS1520+,device_name=mabel,objectIdentifier=1.3.6.1.4.1.6574.1.3.0,instrumentation.name=disk_station,upgradeAvailable=available,systemStatus=normal,systemFanStatus=normal,powerStatus=normal,version=DSM\ 7.0-42218,serialNumber=2110RYRMQ9AXB,tags.type=foo,cpuFanStatus=normal powerStatus=1i 1658869145000000000
kentik.SYNOLOGY-SYSTEM-MIB,objectIdentifier=1.3.6.1.4.1.6574.1.1.0,device_name=mabel,upgradeAvailable=available,tags.type=foo,version=DSM\ 7.0-42218,systemStatus=normal,modelName=DS1520+,device_ip=10.10.0.251,instrumentation.name=disk_station,serialNumber=2110RYRMQ9AXB,systemFanStatus=normal,powerStatus=normal,cpuFanStatus=normal systemStatus=1i 1658869145000000000
kentik.SYNOLOGY-SYSTEM-MIB,modelName=DS1520+,serialNumber=2110RYRMQ9AXB,instrumentation.name=disk_station,upgradeAvailable=available,systemStatus=normal,objectIdentifier=1.3.6.1.4.1.6574.1.5.4.0,systemFanStatus=normal,cpuFanStatus=normal,tags.type=foo,version=DSM\ 7.0-42218,device_ip=10.10.0.251,powerStatus=normal,device_name=mabel upgradeAvailable=1i 1658869145000000000
kentik.SYNOLOGY-SYSTEM-MIB,modelName=DS1520+,version=DSM\ 7.0-42218,systemFanStatus=normal,device_name=mabel,cpuFanStatus=normal,serialNumber=2110RYRMQ9AXB,upgradeAvailable=available,tags.type=foo,device_ip=10.10.0.251,objectIdentifier=1.3.6.1.4.1.6574.1.2.0,systemStatus=normal,powerStatus=normal,instrumentation.name=disk_station temperature=37i 1658869145000000000
kentik.SYNOLOGY-DISK-MIB,device_name=mabel,objectIdentifier=1.3.6.1.4.1.6574.2.1.1.5,instrumentation.name=disk_station,device_ip=10.10.0.251,diskModel=ST12000VN0008-2PH103,tags.type=foo,diskID=Disk\ 4,diskType=SATA,index=.0,mib-table=disk,diskStatus=normal diskStatus=1i 1658869145000000000
kentik.SYNOLOGY-DISK-MIB,index=.0,device_name=mabel,device_ip=10.10.0.251,diskModel=ST12000VN0008-2PH103,diskType=SATA,objectIdentifier=1.3.6.1.4.1.6574.2.1.1.6,instrumentation.name=disk_station,mib-table=disk,diskStatus=normal,tags.type=foo,diskID=Disk\ 4 diskTemperature=37i 1658869145000000000
kentik.SYNOLOGY-DISK-MIB,device_ip=10.10.0.251,mib-table=disk,objectIdentifier=1.3.6.1.4.1.6574.2.1.1.5,instrumentation.name=disk_station,tags.type=foo,diskType=SATA,device_name=mabel,diskModel=ST12000VN0008-2PH103,diskID=Disk\ 5,diskStatus=notinitialized,index=.1 diskStatus=3i 1658869145000000000
kentik.SYNOLOGY-DISK-MIB,mib-table=disk,diskID=Disk\ 5,diskStatus=notinitialized,diskType=SATA,index=.1,instrumentation.name=disk_station,device_ip=10.10.0.251,diskModel=ST12000VN0008-2PH103,tags.type=foo,objectIdentifier=1.3.6.1.4.1.6574.2.1.1.6,device_name=mabel diskTemperature=35i 1658869145000000000
kentik.SYNOLOGY-DISK-MIB,device_ip=10.10.0.251,instrumentation.name=disk_station,tags.type=foo,diskID=Disk\ 1,diskType=SATA,device_name=mabel,diskModel=WD120EFBX-68B0EN0,objectIdentifier=1.3.6.1.4.1.6574.2.1.1.6,index=.2,mib-table=disk,diskStatus=normal diskTemperature=36i 1658869145000000000
```

dpajin-kentik: @i3149, @tdanner  I think we should: 
- remove "objectIdentifier" from the tags and combine all the "metrics" with the same tags into the same line. 
- Also, the measurement is using MIB name, but it should be "Table name". Thinking a bit more on this, maybe at the point it should be "MIB name" + "::" + "Table name" to ensure uniqueness across the board. In this example, the measurement should be "SYNOLOGY-DISK-MIB::diskTable" (based on the profile here: https://github.com/kentik/snmp-profiles/blob/main/profiles/kentik_snmp/synology/disk_station.yml)
- The tag named "mib-table" is redundant, so we shoudl remove it.

For example, these two lines:
```
kentik.SYNOLOGY-DISK-MIB,device_ip=10.10.0.251,mib-table=disk,objectIdentifier=1.3.6.1.4.1.6574.2.1.1.5,instrumentation.name=disk_station,tags.type=foo,diskType=SATA,device_name=mabel,diskModel=ST12000VN0008-2PH103,diskID=Disk\ 5,diskStatus=notinitialized,index=.1 diskStatus=3i 1658869145000000000

kentik.SYNOLOGY-DISK-MIB,mib-table=disk,diskID=Disk\ 5,diskStatus=notinitialized,diskType=SATA,index=.1,instrumentation.name=disk_station,device_ip=10.10.0.251,diskModel=ST12000VN0008-2PH103,tags.type=foo,objectIdentifier=1.3.6.1.4.1.6574.2.1.1.6,device_name=mabel diskTemperature=35i 1658869145000000000
```
has the same set of tags (except objectIdentifier), with the same index from the table, so they should be combined into one line with two metrics, like this:

```
kentik.SYNOLOGY-DISK-MIB::diskTable,device_ip=10.10.0.251,instrumentation.name=disk_station,tags.type=foo,diskType=SATA,device_name=mabel,diskModel=ST12000VN0008-2PH103,diskID=Disk\ 5,diskStatus=notinitialized,index=.1 diskStatus=3i,diskTemperature=35i 1658869145000000000
```

where does "instrumentation.name=disk_station" tag comes from, same for "tags.type=foo"?

tdanner: Dusan is absolutely right - these changes would make the output way more usable.
dpajin-kentik: a few more things: 

- for metrics which are enum type, we would need to put the metric's "string" value, also in the Influx fields and not tags, with the metric's name with the additional suffix "_str". For example:
```
kentik.SYNOLOGY-SYSTEM-MIB,device_ip=10.10.0.251,modelName=DS1520+,device_name=mabel,version=DSM\ 7.0-42218,serialNumber=2110RYRMQ9AXB,tags.type=foo powerStatus=1i,cpuFanStatus=1i,upgradeAvailable=1i,systemStatus=1i,systemFanStatus=1i,powerStatus_string=normal,cpuFanStatus_string=normal,upgradeAvailable_string=available,systemStatus_string=normal,systemFanStatus_string=normal 1658869145000000000
```

- the previous point we should do also with the "metrics_tags" from the SNMP profiles so that they are placed in the InfluxDB line format at tags and not fields

- for metrics which are scalar, as those from SYNOLOGY-SYSTEM-MIB, we should use "MIB Name" as the measurement name, which is done, but we should also combine all the scalar metrics from the same MIB into one line submitted to the InfluxDB. The example for the previous point is describing this requirement


i3149: ```
SYNOLOGY-DISK-MIB::disk,device_ip=10.10.0.251,device_name=mabel,diskID=Disk\ 4,diskModel=ST12000VN0008-2PH103,diskStatus=normal,diskType=SATA,index=.0,instrumentation.name=disk_station,tags.type=foo diskStatus=1i,diskTemperature=36i 1658947212000000000
none,SysLocation=Unknown,SysName=mabel,device_ip=10.10.0.251,device_name=mabel,instrumentation.name=disk_station,modelName=DS1520+,serialNumber=2110RYRMQ9AXB,src_addr=10.10.0.251,tags.type=foo,version=DSM\ 7.0-42218 Uptime=591854i 1658947212000000000
IF-MIB::if,device_name=mabel,if_Address=169.254.11.80,if_AdminStatus=up,if_Description=ovs_eth0,if_Index=8,if_Netmask=255.255.0.0,if_PhysAddress=00:11:32:ee:8f:29,if_Speed=0,if_Type=ethernetCsmacd,if_interface_name=ovs_eth0,instrumentation.name=disk_station,tags.type=foo ifOutDiscards=0i,ifHCOutBroadcastPkts=0i,ifHCInMulticastPkts=0i,ifHCInBroadcastPkts=0i,ifHCOutMulticastPkts=0i,ifInDiscards=0i,if_AdminStatus=1i,ifHCInUcastPkts=0i,ifHCOutUcastPkts=0i 1658947212000000000
IF-MIB::if,device_name=mabel,if_AdminStatus=up,if_Description=eth0,if_Index=5,if_PhysAddress=00:11:32:ee:8f:29,if_Speed=0,if_Type=ethernetCsmacd,if_interface_name=eth0,instrumentation.name=disk_station,tags.type=foo ifHCInUcastPkts=0i,ifInDiscards=0i,ifHCInBroadcastPkts=0i,ifHCInMulticastPkts=0i,ifHCOutMulticastPkts=0i,ifHCOutUcastPkts=0i,ifOutDiscards=0i,ifHCOutBroadcastPkts=0i,if_AdminStatus=1i 1658947212000000000
SYNOLOGY-SYSTEM-MIB,cpuFanStatus=normal,device_ip=10.10.0.251,device_name=mabel,instrumentation.name=disk_station,modelName=DS1520+,powerStatus=normal,serialNumber=2110RYRMQ9AXB,systemFanStatus=normal,systemStatus=normal,tags.type=foo,upgradeAvailable=available,version=DSM\ 7.0-42218 systemStatus=1i,cpuFanStatus=1i,powerStatus=1i,temperature=36i,upgradeAvailable=1i,systemFanStatus=1i 1658947212000000000
```
tdanner: Nice update! I think we can come up with a better measurement name than `none` for the device-level scalars though. @dpajin-kentik recommended the MIB name, which makes sense to me.
i3149: >  "instrumentation.name=disk_station" tag comes from, same for "tags.type=foo"?

`instrumentation.name` is an alias for provider. 

`tags.*` are the tagging system. Users can define tags per device or globally and then these come in as attributes on all metrics. See https://github.com/kentik/ktranslate/pull/358 for where this feature is heading. 

Patch:
diff --git a/pkg/formats/influx/influx.go b/pkg/formats/influx/influx.go
index dd76d78..cffed10 100644
--- a/pkg/formats/influx/influx.go
+++ b/pkg/formats/influx/influx.go
@@ -1,8 +1,10 @@
 package influx
 
 import (
+	"flag"
 	"fmt"
 	"regexp"
+	"sort"
 	"strconv"
 	"strings"
 	"sync"
@@ -30,18 +32,11 @@ type InfluxData struct {
 	Timestamp   int64
 }
 
-func (d *InfluxData) String() string {
-	fields := make([]string, len(d.Fields)+len(d.FieldsFloat))
-	i := 0
-	for k, v := range d.Fields {
-		fields[i] = k + "=" + strconv.FormatInt(v, 10) + "i"
-		i++
-	}
-	for k, v := range d.FieldsFloat {
-		fields[i] = k + "=" + strconv.FormatFloat(v, 'f', 4, 64)
-		i++
-	}
+var (
+	Prefix = flag.String("influxdb_measurement_prefix", "", "Prefix metric names with this")
+)
 
+func (d *InfluxData) GetTags() string {
 	var tags []string
 	for key, v := range d.Tags {
 		kval := strings.ReplaceAll(key, " ", "_")
@@ -55,19 +50,95 @@ func (d *InfluxData) String() string {
 		}
 	}
 
+	sort.Strings(tags) // Need to be sorted for dedupe.
+	return strings.Join(tags, ",")
+}
+
+func (d *InfluxData) Prefix() string {
+	return fmt.Sprintf("%s,%s",
+		d.Name,
+		d.GetTags(),
+	)
+}
+
+func (d *InfluxData) String() string {
+	fields := make([]string, len(d.Fields)+len(d.FieldsFloat))
+	i := 0
+	for k, v := range d.Fields {
+		if ev, ok := d.Tags[k]; ok { // There's an enum here, use this vs the int value.
+			fields[i] = k + "=" + ev.(string)
+			delete(d.Tags, k)
+		} else {
+			fields[i] = k + "=" + strconv.FormatInt(v, 10) + "i"
+		}
+		i++
+	}
+	for k, v := range d.FieldsFloat {
+		if ev, ok := d.Tags[k]; ok { // There's an enum here, use this vs the int value.
+			fields[i] = k + "=" + ev.(string)
+			delete(d.Tags, k)
+		} else {
+			fields[i] = k + "=" + strconv.FormatFloat(v, 'f', 4, 64)
+		}
+		i++
+	}
+
 	return fmt.Sprintf("%s,%s %s %d",
 		d.Name,
-		strings.Join(tags, ","),
+		d.GetTags(),
 		strings.Join(fields, ","),
 		d.Timestamp)
 }
 
+func NewMergedInfluxData(s InfluxDataSet) *InfluxData {
+	if len(s) == 0 {
+		return nil
+	}
+	d := InfluxData{
+		Name:        s[0].Name,
+		Tags:        s[0].Tags,
+		Timestamp:   s[0].Timestamp,
+		FieldsFloat: map[string]float64{},
+		Fields:      map[string]int64{},
+	}
+
+	for _, f := range s {
+		for k, v := range f.FieldsFloat {
+			d.FieldsFloat[k] = v
+		}
+		for k, v := range f.Fields {
+			d.Fields[k] = v
+		}
+	}
+
+	return &d
+}
+
 type InfluxDataSet []InfluxData
 
 func (s InfluxDataSet) Bytes() []byte {
-	res := make([]string, 0)
+	// First map common prefixes.
+	prefixes := map[string][]InfluxData{}
 	for _, l := range s {
-		res = append(res, l.String())
+		prefix := l.Prefix()
+		if _, ok := prefixes[prefix]; !ok {
+			prefixes[prefix] = []InfluxData{}
+		}
+		prefixes[prefix] = append(prefixes[prefix], l)
+	}
+
+	// Now merge down any common prefixes.
+	merged := []*InfluxData{}
+	for _, l := range prefixes {
+		merged = append(merged, NewMergedInfluxData(l))
+	}
+
+	// Then format for output.
+	res := make([]string, 0)
+	for _, l := range merged {
+		if l != nil {
+			res = append(res, l.String())
+		}
 	}
 	return []byte(strings.Join(res, "\n"))
 }
@@ -181,7 +252,7 @@ func (f *InfluxFormat) fromKSynth(in *kt.JCHF) []InfluxData {
 	}
 
 	return []InfluxData{InfluxData{
-		Name:      "kentik.synth",
+		Name:      *Prefix,
 		Fields:    ms,
 		Timestamp: in.Timestamp * 1000000000,
 		Tags:      attr,
@@ -212,7 +283,7 @@ func (f *InfluxFormat) fromKflow(in *kt.JCHF) []InfluxData {
 	}
 
 	return []InfluxData{InfluxData{
-		Name:      "kentik.flow",
+		Name:      *Prefix,
 		Fields:    ms,
 		Timestamp: in.Timestamp * 1000000000,
 		Tags:      attr,
@@ -225,6 +296,7 @@ func (f *InfluxFormat) fromSnmpDeviceMetric(in *kt.JCHF) []InfluxData {
 	f.mux.RLock()
 	util.SetAttr(attr, in, metrics, f.lastMetadata[in.DeviceName])
 	f.mux.RUnlock()
+	ip := attr["src_addr"]
 
 	results := []InfluxData{}
 	for m, name := range metrics {
@@ -238,17 +310,17 @@ func (f *InfluxFormat) fromSnmpDeviceMetric(in *kt.JCHF) []InfluxData {
 				continue // This Metric isn't in the white list so lets drop it.
 			}
 
-			mtype := name.GetType()
+			mib := getMib(attrNew, ip)
 			if name.Format == kt.FloatMS {
 				results = append(results, InfluxData{
-					Name:        "kentik." + mtype,
+					Name:        *Prefix + mib,
 					FieldsFloat: map[string]float64{m: float64(float64(in.CustomBigInt[m]) / 1000)},
 					Timestamp:   in.Timestamp * 1000000000,
 					Tags:        attrNew,
 				})
 			} else {
 				results = append(results, InfluxData{
-					Name:      "kentik." + mtype,
+					Name:      *Prefix + mib,
 					Fields:    map[string]int64{m: int64(in.CustomBigInt[m])},
 					Timestamp: in.Timestamp * 1000000000,
 					Tags:      attrNew,
@@ -266,6 +338,7 @@ func (f *InfluxFormat) fromSnmpInterfaceMetric(in *kt.JCHF) []InfluxData {
 	f.mux.RLock()
 	defer f.mux.RUnlock()
 	util.SetAttr(attr, in, metrics, f.lastMetadata[in.DeviceName])
+	ip := attr["src_ip"]
 
 	profileName := "snmp"
 	results := []InfluxData{}
@@ -281,17 +354,17 @@ func (f *InfluxFormat) fromSnmpInterfaceMetric(in *kt.JCHF) []InfluxData {
 				continue // This Metric isn't in the white list so lets drop it.
 			}
 
-			mtype := name.GetType()
+			mib := getMib(attrNew, ip)
 			if name.Format == kt.FloatMS {
 				results = append(results, InfluxData{
-					Name:        "kentik." + mtype,
+					Name:        *Prefix + mib,
 					FieldsFloat: map[string]float64{m: float64(float64(in.CustomBigInt[m]) / 1000)},
 					Timestamp:   in.Timestamp * 1000000000,
 					Tags:        attrNew,
 				})
 			} else {
 				results = append(results, InfluxData{
-					Name:      "kentik." + mtype,
+					Name:      *Prefix + mib,
 					Fields:    map[string]int64{m: int64(in.CustomBigInt[m])},
 					Timestamp: in.Timestamp * 1000000000,
 					Tags:      attrNew,
@@ -311,7 +384,7 @@ func (f *InfluxFormat) fromSnmpInterfaceMetric(in *kt.JCHF) []InfluxData {
 						if inBytes, ok := in.CustomBigInt["ifHCInOctets"]; ok {
 							if !util.DropOnFilter(attrNew, f.lastMetadata[in.DeviceName], true) {
 								results = append(results, InfluxData{
-									Name:        "kentik.snmp",
+									Name:        *Prefix + "if",
 									FieldsFloat: map[string]float64{"IfInUtilization": float64(inBytes*8*100) / float64(uptimeSpeed)},
 									Timestamp:   in.Timestamp * 1000000000,
 									Tags:        attrNew,
@@ -331,7 +404,7 @@ func (f *InfluxFormat) fromSnmpInterfaceMetric(in *kt.JCHF) []InfluxData {
 						if outBytes, ok := in.CustomBigInt["ifHCOutOctets"]; ok {
 							if !util.DropOnFilter(attrNew, f.lastMetadata[in.DeviceName], true) {
 								results = append(results, InfluxData{
-									Name:        "kentik.snmp",
+									Name:        *Prefix + "if",
 									FieldsFloat: map[string]float64{"IfOutUtilization": float64(outBytes*8*100) / float64(uptimeSpeed)},
 									Timestamp:   in.Timestamp * 1000000000,
 									Tags:        attrNew,
@@ -347,13 +420,58 @@ func (f *InfluxFormat) fromSnmpInterfaceMetric(in *kt.JCHF) []InfluxData {
 	return results
 }
 
-var escaper = regexp.MustCompile("([,= ])")
+var escaper = regexp.MustCompile("([,= \\s])")
 
 // Escape special characters according to https://docs.influxdata.com/influxdb/v1.8/write_protocols/line_protocol_tutorial/#special-characters-and-keywords
 func influxEscape(s string) string {
-	if strings.ContainsAny(s, ",= ") {
+	if strings.ContainsAny(s, ",= \t\r\n") {
 		return string(escaper.ReplaceAll([]byte(s), []byte("\\$1")))
 	} else {
 		return s
 	}
 }
+
+func getMib(attr map[string]interface{}, ip interface{}) string {
+	// Remove any lingering droppable fields.
+	for k, v := range attr {
+		if _, ok := dropFields[k]; ok {
+			delete(attr, k)
+		}
+		if k == "Index" {
+			delete(attr, k)
+			attr["index"] = v
+		}
+	}
+	if ip != nil {
+		attr["device_ip"] = ip
+	}
+
+	// And now figure out what the mib name is.
+	mib, ok := attr["mib-name"].(string)
+	if !ok {
+		return "device"
+	}
+	delete(attr, "mib-name")
+	if mib == "" {
+		mib = "device"
+	}
+
+	// If there's a table, add this to the info.
+	mibTable, ok := attr["mib-table"].(string)
+	if ok {
+		mib = mib + "::" + mibTable
+		delete(attr, "mib-table")
+	}
+
+	return mib
+}
+
+var dropFields = map[string]bool{
+	//"device_name": true,
+	"objectIdentifier": true,
+	"eventType":        true,
+	"SysObjectID":      true,
+	"provider":         true,
+	"SysServices":      true,
+	"SysDescr":         true,
+}
diff --git a/pkg/inputs/snmp/snmp.go b/pkg/inputs/snmp/snmp.go
index 426b68e..dd46036 100644
--- a/pkg/inputs/snmp/snmp.go
+++ b/pkg/inputs/snmp/snmp.go
@@ -204,6 +204,9 @@ func runSnmpPolling(ctx context.Context, snmpFile string, jchfChan chan []*kt.JC
 			}
 		}
 
+		// Tweak any per provider tags and match attributes here.
+		setDeviceTagsAndMatch(device)
+
 		// Create this device in Kentik if the option is set.
 		err := apic.EnsureDevice(ctx, device)
 		if err != nil {
@@ -371,6 +374,10 @@ func parseConfig(ctx context.Context, file string, log logger.ContextL) (*kt.Snm
 
 	// If there's a global user tags and match, add them in here.
 	if ms.Global != nil {
+		for p, m := range ms.Global.ProviderMap {
+			m.Init(p, &ms) // Set up any provider based user and match tags here.
+		}
+
 		for k, v := range ms.Global.UserTags {
 			for _, device := range ms.Devices {
 				if device.UserTags == nil {
@@ -435,3 +442,58 @@ func launchExtOnly(ctx context.Context, conf *kt.SnmpGlobalConfig, device *kt.Sn
 func ParseConfig(file string, log logger.ContextL) (*kt.SnmpConfig, error) {
 	return parseConfig(context.Background(), file, log)
 }
+
+// If there's a provider: prefix here
+func matchesPrefix(tag string, provider kt.Provider) (string, bool) {
+	if !strings.HasPrefix(tag, kt.ProviderPrefix) { // No prefix so just return true.
+		return tag, true
+	}
+
+	pts := strings.SplitN(tag, kt.ProviderToken, 3)
+	if len(pts) < 3 { // Invalid prefix, just return true here.
+		return tag, true
+	}
+
+	if pts[1] == string(provider) {
+		return pts[2], true
+	}
+
+	return "", false
+}
+
+func setDeviceTagsAndMatch(device *kt.SnmpDeviceConfig) {
+	set := func(m map[string]string, p kt.Provider) {
+		for k, v := range m {
+			if nk, ok := matchesPrefix(k, p); ok {
+				delete(m, k)
+				if _, ok := m[nk]; !ok { // Don't overwrite an existing key.
+					m[nk] = v
+				}
+			}
+		}
+	}
+
+	prune := func(m map[string]string) {
+		for k, _ := range m {
+			if strings.HasPrefix(k, kt.ProviderPrefix) {
+				delete(m, k)
+			}
+		}
+	}
+
+	// First fill in any device level sets.
+	set(device.UserTags, kt.DeviceProvider)
+	set(device.MatchAttr, kt.DeviceProvider)
+
+	// Then any provider level
+	set(device.UserTags, device.Provider)
+	set(device.MatchAttr, device.Provider)
+
+	// Lastly any global
+	set(device.UserTags, kt.GlobalProvider)
+	set(device.MatchAttr, kt.GlobalProvider)
+
+	// Now take out any unset keys.
+	prune(device.UserTags)
+	prune(device.MatchAttr)
+}
diff --git a/pkg/kt/snmp.go b/pkg/kt/snmp.go
index dfb468a..c2539b5 100644
--- a/pkg/kt/snmp.go
+++ b/pkg/kt/snmp.go
@@ -16,7 +16,11 @@ import (
 )
 
 const (
-	UserTagPrefix = "tags."
+	UserTagPrefix  = "tags."
+	ProviderToken  = ":"
+	ProviderPrefix = "provider" + ProviderToken
+	GlobalProvider = "global"
+	DeviceProvider = "device"
 )
 
 // DeviceData holds information about a device, sent via ST, and sent
@@ -218,20 +222,26 @@ type SnmpDiscoConfig struct {
 	IgnoreOrig         string          `yaml:"-"`
 }
 
+type ProviderMap struct {
+	UserTags  map[string]string `yaml:"user_tags"`
+	MatchAttr map[string]string `yaml:"match_attributes"`
+}
+
 type SnmpGlobalConfig struct {
-	PollTimeSec   int               `yaml:"poll_time_sec"`
-	DropIfOutside bool              `yaml:"drop_if_outside_poll"`
-	MibProfileDir string            `yaml:"mib_profile_dir"`
-	MibDB         string            `yaml:"mibs_db"`
-	MibsEnabled   []string          `yaml:"mibs_enabled"`
-	TimeoutMS     int               `yaml:"timeout_ms"`
-	Retries       int               `yaml:"retries"`
-	GlobalV3      *V3SNMPConfig     `yaml:"global_v3"`
-	RunPing       bool              `yaml:"response_time"`
-	PingSec       int               `yaml:"ping_interval_sec,omitempty"`
-	PurgeDevices  int               `yaml:"purge_devices_after_num"` // Delete any device if its not seen after X discovery attempts. Default is 0, which means things never get purged.
-	UserTags      map[string]string `yaml:"user_tags"`
-	MatchAttr     map[string]string `yaml:"match_attributes"`
+	PollTimeSec   int                    `yaml:"poll_time_sec"`
+	DropIfOutside bool                   `yaml:"drop_if_outside_poll"`
+	MibProfileDir string                 `yaml:"mib_profile_dir"`
+	MibDB         string                 `yaml:"mibs_db"`
+	MibsEnabled   []string               `yaml:"mibs_enabled"`
+	TimeoutMS     int                    `yaml:"timeout_ms"`
+	Retries       int                    `yaml:"retries"`
+	GlobalV3      *V3SNMPConfig          `yaml:"global_v3"`
+	RunPing       bool                   `yaml:"response_time"`
+	PingSec       int                    `yaml:"ping_interval_sec,omitempty"`
+	PurgeDevices  int                    `yaml:"purge_devices_after_num"` // Delete any device if its not seen after X discovery attempts. Default is 0, which means things never get purged.
+	UserTags      map[string]string      `yaml:"user_tags"`
+	MatchAttr     map[string]string      `yaml:"match_attributes"`
+	ProviderMap   map[string]ProviderMap `yaml:"providers"`
 }
 
 type SnmpConfig struct {
@@ -240,6 +250,7 @@ type SnmpConfig struct {
 	Disco      *SnmpDiscoConfig  `yaml:"discovery"`
 	Global     *SnmpGlobalConfig `yaml:"global"`
 	DeviceOrig string            `yaml:"-"`
+	doneInit   bool              `yaml:"-"`
 }
 
 type SnmpMetricSet struct {
@@ -704,3 +715,50 @@ func (a *EAPIConfig) UnmarshalYAML(unmarshal func(interface{}) error) error {
 
 	return nil
 }
+
+func (p *ProviderMap) Init(provider string, cfg *SnmpConfig) {
+	if cfg.Global == nil {
+		cfg.Global = &SnmpGlobalConfig{}
+	}
+	g := cfg.Global
+
+	if p.UserTags != nil && g.UserTags == nil {
+		g.UserTags = map[string]string{}
+	}
+	if p.MatchAttr != nil && g.MatchAttr == nil {
+		g.MatchAttr = map[string]string{}
+	}
+
+	// Set up a global provider for global tags.
+	if !cfg.doneInit {
+		cfg.doneInit = true
+		for k, v := range g.UserTags {
+			delete(g.UserTags, k)
+			g.UserTags[ProviderPrefix+GlobalProvider+ProviderToken+k] = v
+		}
+		for k, v := range g.MatchAttr {
+			delete(g.MatchAttr, k)
+			g.MatchAttr[ProviderPrefix+GlobalProvider+ProviderToken+k] = v
+		}
+
+		// Set up device prefix for device level ones.
+		for _, device := range cfg.Devices {
+			for k, v := range device.UserTags {
+				delete(device.UserTags, k)
+				device.UserTags[ProviderPrefix+DeviceProvider+ProviderToken+k] = v
+			}
+			for k, v := range device.MatchAttr {
+				delete(device.MatchAttr, k)
+				device.MatchAttr[ProviderPrefix+DeviceProvider+ProviderToken+k] = v
+			}
+		}
+	}
+
+	// Copy these over in the right order to get processed by the regular per device system.
+	for k, v := range p.UserTags {
+		g.UserTags[ProviderPrefix+provider+ProviderToken+k] = v
+	}
+	for k, v := range p.MatchAttr {
+		g.MatchAttr[ProviderPrefix+provider+ProviderToken+k] = v
+	}
+}


Test patch:
diff --git a/pkg/formats/influx/influx_test.go b/pkg/formats/influx/influx_test.go
index c57858d..e968c08 100644
--- a/pkg/formats/influx/influx_test.go
+++ b/pkg/formats/influx/influx_test.go
@@ -37,6 +37,8 @@ func TestInfluxEscape(t *testing.T) {
 		{"as,df", "as\\,df"},
 		{"as=df", "as\\=df"},
 		{"as, df", "as\\,\\ df"},
+		{"as\ndf", "as\\\ndf"},
+		{"as\r\ndf", "as\\\r\\\ndf"},
 	}
 	for _, test := range tests {
 		if got := influxEscape(test.input); got != test.want {
diff --git a/pkg/inputs/snmp/snmp_test.go b/pkg/inputs/snmp/snmp_test.go
new file mode 100644
index 0000000..9032e6f
--- /dev/null
+++ b/pkg/inputs/snmp/snmp_test.go
@@ -0,0 +1,179 @@
+package snmp
+
+import (
+	"fmt"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+
+	"github.com/kentik/ktranslate/pkg/kt"
+)
+
+func TestMatchesPrefix(t *testing.T) {
+	assert := assert.New(t)
+
+	tests := map[string][]string{
+		"tagName":               []string{"tagName", "true"},
+		"provider:tagName":      []string{"provider:tagName", "true"},
+		"provider:foo:tagName":  []string{"tagName", "true"},
+		"provider:bar:tagName":  []string{"", "false"},
+		"provider:foo:tagName1": []string{"tagName1", "true"},
+	}
+
+	provider := kt.Provider("foo")
+
+	for in, expt := range tests {
+		newTag, res := matchesPrefix(in, provider)
+		assert.Equal(expt[0], newTag, "%s <-> %s", in, provider)
+		assert.Equal(expt[1], fmt.Sprintf("%v", res), "%s <-> %s", in, provider)
+	}
+}
+
+func TestSetTagsMatch(t *testing.T) {
+	assert := assert.New(t)
+
+	tests := map[string]kt.SnmpConfig{
+		"one": kt.SnmpConfig{
+			Global: &kt.SnmpGlobalConfig{
+				UserTags: map[string]string{
+					"tag": "global",
+				},
+				MatchAttr: map[string]string{
+					"match": "global",
+				},
+				ProviderMap: map[string]kt.ProviderMap{
+					"foo": kt.ProviderMap{
+						UserTags: map[string]string{
+							"tag": "provider",
+						},
+						MatchAttr: map[string]string{
+							"match": "provider",
+						},
+					},
+					"bar": kt.ProviderMap{
+						UserTags: map[string]string{
+							"tag": "provider",
+						},
+						MatchAttr: map[string]string{
+							"match": "provider",
+						},
+					},
+				},
+			},
+			Devices: map[string]*kt.SnmpDeviceConfig{
+				"device": &kt.SnmpDeviceConfig{
+					Provider: "foo",
+					UserTags: map[string]string{
+						"tag": "device", // This should be device tag because its set at device level.
+					},
+					MatchAttr: map[string]string{
+						"match": "device",
+					},
+				},
+				"provider": &kt.SnmpDeviceConfig{
+					Provider: "foo",
+					UserTags: map[string]string{
+						"tagA": "device", // This should fall back to provider level.
+					},
+					MatchAttr: map[string]string{
+						"matchA": "device",
+					},
+				},
+				"global": &kt.SnmpDeviceConfig{
+					Provider: "fooA",
+					UserTags: map[string]string{
+						"tagA": "device", // This should fall back to global level because niether provider or device set.
+					},
+					MatchAttr: map[string]string{
+						"matchA": "device",
+					},
+				},
+			},
+		},
+		"two": kt.SnmpConfig{ // No provider, just gobal and device.
+			Global: &kt.SnmpGlobalConfig{
+				UserTags: map[string]string{
+					"tag": "global",
+				},
+				MatchAttr: map[string]string{
+					"match": "global",
+				},
+			},
+			Devices: map[string]*kt.SnmpDeviceConfig{
+				"device": &kt.SnmpDeviceConfig{
+					Provider: "foo",
+					UserTags: map[string]string{
+						"tag": "device", // This should be device tag because its set at device level.
+					},
+					MatchAttr: map[string]string{
+						"match": "device",
+					},
+				},
+				"global": &kt.SnmpDeviceConfig{
+					Provider: "fooA",
+					UserTags: map[string]string{
+						"tagA": "device", // This should fall back to global level because niether provider or device set.
+					},
+					MatchAttr: map[string]string{
+						"matchA": "device",
+					},
+				},
+			},
+		},
+		"three": kt.SnmpConfig{ // No provider, just gobal and device.
+			Global: &kt.SnmpGlobalConfig{},
+			Devices: map[string]*kt.SnmpDeviceConfig{
+				"device": &kt.SnmpDeviceConfig{
+					Provider: "foo",
+					UserTags: map[string]string{
+						"tag": "device", // This should be device tag because its set at device level.
+					},
+					MatchAttr: map[string]string{
+						"match": "device",
+					},
+				},
+				"": &kt.SnmpDeviceConfig{
+					Provider: "fooA",
+					UserTags: map[string]string{
+						"tagA": "device", // This should fall back to global level because niether provider or device set.
+					},
+					MatchAttr: map[string]string{
+						"matchA": "device",
+					},
+				},
+			},
+		},
+	}
+
+	for test, ms := range tests {
+		for p, m := range ms.Global.ProviderMap {
+			m.Init(p, &ms) // Set up any provider based user and match tags here.
+		}
+		for k, v := range ms.Global.UserTags {
+			for _, device := range ms.Devices {
+				if device.UserTags == nil {
+					device.UserTags = map[string]string{}
+				}
+				if _, ok := device.UserTags[k]; !ok {
+					device.UserTags[k] = v
+				}
+			}
+		}
+		for k, v := range ms.Global.MatchAttr {
+			for _, device := range ms.Devices {
+				if device.MatchAttr == nil {
+					device.MatchAttr = map[string]string{}
+				}
+				if _, ok := device.MatchAttr[k]; !ok {
+					device.MatchAttr[k] = v
+				}
+			}
+		}
+
+		for expt, device := range ms.Devices {
+			setDeviceTagsAndMatch(device)
+			assert.Equal(expt, device.UserTags["tag"], "%s -> %s %v", test, device.Provider, device.UserTags)
+			assert.Equal(expt, device.MatchAttr["match"], "%s -> %s %v", test, device.Provider, device.MatchAttr)
+		}
+	}
+}