Repository: kentik/ktranslate (go)

Issue:
[Feature Request] Add additional configuration items to filter Netbox devices
Hi team,

First off, thank you for the excellent work on ktranslate—it’s been instrumental in automating our SNMP-based discovery pipeline.

We’re currently using the netbox integration in the discovery section of the SNMP configuration, and it works well for basic filtering by site and ip_to_pick. However, as our NetBox instance becomes more richly populated and structured, we’d greatly benefit from enhanced filtering capabilities.

Feature Request

I would like to request support for additional filtering options in the netbox configuration block of snmp-base.yaml:
	•	tenant – Filter devices based on assigned tenants
	•	role – Filter devices based on device roles
	•	location – Filter by physical or logical location
	•	status – Filter based on device operational status (e.g., active, staged, decommissioned)
	•	custom_fields – Filter devices using custom key/value metadata defined in NetBox

Use Case

This added filtering flexibility would allow us to:
	•	Limit SNMP polling to only active or production-ready devices
	•	Segment polling responsibilities across tenants or business units
	•	Selectively monitor devices based on logical roles (e.g., only “edge-routers”)
	•	Support more granular automation scenarios, especially in multi-tenant environments
	•	Fully leverage NetBox as our single source of truth

Proposed Configuration Example
```
netbox:
  host: http://netbox.default.svc.cluster.local
  token: <redacted>
  site: Gothenburg
  tenant: infra
  role: edge-router
  location: dc1
  status: active
  custom_fields:
    monitor: true
  ip_to_pick: oob
```

This would dramatically reduce the need for manual device filtering logic outside of ktranslate and improve operational clarity.

Please let me know if this is something on the roadmap or if there’s any way I can help contribute to the feature.

Thanks again for your time and the great tooling!

Pull request description:
Netbox oauth
Closes #812 

This is a reworking of how netbox integration works. A new config will look like 

```
    netbox:
        url: https://fcmd3592.cloud.netboxapp.com/api/dcim/devices
        token: ${NETBOX_TOKEN}
        ip_to_pick: primary
        tag: foxtrot
        site: sydney
        location: ""
        tenant: ""
        status: active
        role: access-switch
        custom_fields:
            my_test: fooo-test
```

You can also set `KTRANS_OAUTH_CLIENT_ID`, `KTRANS_OAUTH_CLIENT_SECRET`, `KTRANS_OAUTH_SCOPE` and `KTRANS_OAUTH_TOKEN_URL` environment variables to use andoauth system to get a bearer token for netbox. In this case, leave the token field in the config empty. 

Note that `host` has been replaced with `url`. This is to accommodate different url patterns for this same api call. 

All of the filters are evaluated server side now except for custom_fields. 

Patch:
diff --git a/go.mod b/go.mod
index 3be5bf8..f0f03f7 100644
--- a/go.mod
+++ b/go.mod
@@ -39,7 +39,6 @@ require (
 	github.com/liamg/furious v0.0.0-20191231090757-c295c872d6c1
 	github.com/linkedin/goavro/v2 v2.10.1
 	github.com/montanaflynn/stats v0.7.0
-	github.com/netbox-community/go-netbox/v4 v4.2.2-3
 	github.com/netsampler/goflow2/v2 v2.1.3
 	github.com/oschwald/geoip2-golang v1.9.0
 	github.com/pkg/errors v0.9.1
diff --git a/go.sum b/go.sum
index 346ee45..9b9175e 100644
--- a/go.sum
+++ b/go.sum
@@ -732,8 +732,6 @@ github.com/montanaflynn/stats v0.7.0/go.mod h1:etXPPgVO6n31NxCd9KQUMvCM+ve0ruNzt
 github.com/mostlygeek/arp v0.0.0-20170424181311-541a2129847a h1:AfneHvfmYgUIcgdUrrDFklLdEzQAvG9AKRTe1x1mx/0=
 github.com/mostlygeek/arp v0.0.0-20170424181311-541a2129847a/go.mod h1:jZxafo9CAqaKFQE4zitrg5QNlA6CXUsjwXPlIppF3tk=
 github.com/ncw/swift v1.0.52/go.mod h1:23YIA4yWVnGwv2dQlN4bB7egfYX6YLn0Yo/S6zZO/ZM=
-github.com/netbox-community/go-netbox/v4 v4.2.2-3 h1:NrAkuI41ExiYa6p1o7F0NvxcS2qxsVJfymR5hAZglDs=
-github.com/netbox-community/go-netbox/v4 v4.2.2-3/go.mod h1:X7jbSuzejM0U5uy/ajXidRFEAshl6eKqrcinhMy4aAI=
 github.com/netsampler/goflow2/v2 v2.1.3 h1:glfeG2hIzFlAmEz236nZIAWi848AB+p1pJnuQqiyLkI=
 github.com/netsampler/goflow2/v2 v2.1.3/go.mod h1:94ZaxfHuUwG6KviCxMWuZIvz0UGu657StE/g83hlS8A=
 github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno=
diff --git a/pkg/inputs/snmp/disco.go b/pkg/inputs/snmp/disco.go
index 67dde20..08ff22c 100644
--- a/pkg/inputs/snmp/disco.go
+++ b/pkg/inputs/snmp/disco.go
@@ -93,8 +93,7 @@ func Discover(ctx context.Context, log logger.ContextL, pollDuration time.Durati
 	}
 
 	foundDevices := map[string]*kt.SnmpDeviceConfig{}
-	if conf.Disco.Netbox != nil && conf.Disco.Netbox.NetboxAPIHost != "" {
-		log.Infof("Discovering devices from Netbox with tag [%s] and site [%s]", conf.Disco.Netbox.NetboxTag, conf.Disco.Netbox.NetboxSite)
+	if conf.Disco.Netbox != nil {
 		err = getDevicesFromNetbox(ctx, ctl, foundDevices, mdb, conf, kentikDevices, log, ignoreMap)
 		if err != nil {
 			return nil, err
diff --git a/pkg/inputs/snmp/netbox.go b/pkg/inputs/snmp/netbox.go
index e216dd1..f3ae581 100644
--- a/pkg/inputs/snmp/netbox.go
+++ b/pkg/inputs/snmp/netbox.go
@@ -4,11 +4,16 @@ package snmp
 Use netbox API to pull down list of devices to target. Get all devices matching a given tag.
 */
 import (
+	"bytes"
 	"context"
+	"encoding/json"
 	"fmt"
+	"io"
 	"net"
+	"net/http"
 	"net/netip"
 	"net/url"
+	"os"
 	"strconv"
 	"sync"
 	"time"
@@ -18,56 +23,271 @@ import (
 	"github.com/kentik/ktranslate/pkg/kt"
 
 	"github.com/liamg/furious/scan"
-	"github.com/netbox-community/go-netbox/v4"
 )
 
+var (
+	tokenExp    time.Time
+	accessToken string
+)
+
+const (
+	authHeaderName      = "Authorization"
+	authHeaderFormat    = "Token %v"
+	bearerHeaderFormat  = "Bearer %v"
+	languageHeaderName  = "Accept-Language"
+	languageHeaderValue = "en-US"
+)
+
+type OauthResp struct {
+	AccessToken string `json:"access_token"`
+	ExpiresIn   int    `json:"expires_in"`
+}
+
+type NBRespOK struct {
+	Count    int        `json:"count"`
+	Next     *string    `json:"next"`
+	Previous *string    `json:"previous"`
+	Results  []NBResult `json:"results"`
+}
+
+type NBResult struct {
+	ID           int                    `json:"id"`
+	Url          *string                `json:"url"`
+	Name         *string                `json:"name"`
+	DeviceType   *NBDeviceType          `json:"device_type"`
+	PrimaryIp    *NBIP                  `json:"primary_ip"`
+	PrimaryIpv4  *NBIP                  `json:"primary_ip4"`
+	PrimaryIpv6  *NBIP                  `json:"primary_ip6"`
+	OobIp        *NBIP                  `json:"oob_ip"`
+	CustomFields map[string]interface{} `json:"custom_fields"`
+}
+
+type NBDeviceType struct {
+	Name  *string `json:"name"`
+	Model *string `json:"model"`
+}
+
+type NBIP struct {
+	Address *string `json:"address"`
+}
+
+func (i *NBIP) GetVal() string {
+	if i == nil || i.Address == nil {
+		return "<nil>"
+	}
+	return *i.Address
+}
+
+func getToken(oauthTokenUrl string) (string, error) {
+	if time.Now().Before(tokenExp) {
+		return accessToken, nil
+	}
+
+	client_id := os.Getenv("KTRANS_OAUTH_CLIENT_ID")
+	if client_id == "" {
+		return "", fmt.Errorf("Set envroment variable KTRANS_OAUTH_CLIENT_ID")
+	}
+	client_secret := os.Getenv("KTRANS_OAUTH_CLIENT_SECRET")
+	if client_secret == "" {
+		return "", fmt.Errorf("Set envroment variable KTRANS_OAUTH_CLIENT_SECRET")
+	}
+	client_scope := os.Getenv("KTRANS_OAUTH_SCOPE")
+	if client_scope == "" {
+		return "", fmt.Errorf("Set envroment variable KTRANS_OAUTH_SCOPE")
+	}
+
+	// Load up the url encoded payload here.
+	payload := url.Values{}
+	payload.Set("grant_type", "client_credentials")
+	payload.Set("client_id", client_id)
+	payload.Set("client_secret", client_secret)
+	payload.Set("scope", client_scope)
+
+	resp, err := http.Post(oauthTokenUrl, "application/x-www-form-urlencoded", bytes.NewBufferString(payload.Encode()))
+	if err != nil {
+		return "", err
+	}
+	body, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return "", err
+	}
+
+	if resp.StatusCode == http.StatusOK {
+		var res OauthResp
+		err = json.Unmarshal(body, &res)
+		if err != nil {
+			return "", err
+		}
+
+		accessToken = res.AccessToken
+		tokenExp = time.Now().Add(time.Duration(res.ExpiresIn-60) * time.Second)
+		return accessToken, nil
+	} else {
+		return "", fmt.Errorf("Invalid response from oauth server: %v %v", resp.StatusCode, string(body))
+	}
+}
+
+func setupDcimFilter(conf *kt.SnmpConfig, log logger.ContextL, offset int32, limit int32) url.Values {
+	// Set up some url filters here.
+	v := url.Values{}
+	v.Add("limit", fmt.Sprintf("%d", limit))
+	v.Add("offset", fmt.Sprintf("%d", offset))
+	v.Add("interface_count__gt", "0")
+
+	if conf.Disco.Netbox.Tag != "" {
+		log.Infof("Adding netbox filter for tag %s", conf.Disco.Netbox.Tag)
+		v.Add("tag", conf.Disco.Netbox.Tag)
+	}
+	if conf.Disco.Netbox.Site != "" {
+		log.Infof("Adding netbox filter for site %s", conf.Disco.Netbox.Site)
+		v.Add("site", conf.Disco.Netbox.Site)
+	}
+	if conf.Disco.Netbox.Tenant != "" {
+		log.Infof("Adding netbox filter for tenant %s", conf.Disco.Netbox.Tenant)
+		v.Add("tenant", conf.Disco.Netbox.Tenant)
+	}
+	if conf.Disco.Netbox.Status != "" {
+		log.Infof("Adding netbox filter for status %s", conf.Disco.Netbox.Status)
+		v.Add("status", conf.Disco.Netbox.Status)
+	} else { // Default to active.
+		log.Infof("Adding netbox filter for status active")
+		v.Add("status", "active")
+	}
+	if conf.Disco.Netbox.Role != "" {
+		log.Infof("Adding netbox filter for role %s", conf.Disco.Netbox.Role)
+		v.Add("role", conf.Disco.Netbox.Role)
+	}
+	if conf.Disco.Netbox.Location != "" {
+		log.Infof("Adding netbox filter for location %s", conf.Disco.Netbox.Location)
+		v.Add("location", conf.Disco.Netbox.Location)
+	}
+
+	return v
+}
+
+func getDcimDevicesApi(ctx context.Context, conf *kt.SnmpConfig, log logger.ContextL, offset int32, limit int32) (*NBRespOK, error) {
+
+	var url *url.URL
+	if conf.Disco.Netbox.NetboxAPIUrl != "" {
+		u, err := url.Parse(conf.Disco.Netbox.NetboxAPIUrl)
+		if err != nil {
+			return nil, err
+		}
+		url = u
+	} else {
+		return nil, fmt.Errorf("Missing url for netbox.")
+	}
+	url.RawQuery = setupDcimFilter(conf, log, offset, limit).Encode()
+	log.Infof("Calling netbox at %s", url.String())
+
+	client := &http.Client{}
+
+	req, err := http.NewRequestWithContext(ctx, "GET", url.String(), nil)
+	if err != nil {
+		return nil, err
+	}
+
+	req.Header.Set("Content-Type", "application/json")
+	if conf.Disco.Netbox.NetboxAPIToken.String() == "" {
+		oauthTokenUrl := os.Getenv("KTRANS_OAUTH_TOKEN_URL")
+		if oauthTokenUrl != "" {
+			log.Infof("Trying to get bearer token from %s", oauthTokenUrl)
+			t, err := getToken(oauthTokenUrl)
+			if err != nil {
+				return nil, err
+			}
+			req.Header.Set(authHeaderName, fmt.Sprintf(bearerHeaderFormat, t))
+		} else {
+			log.Infof("Skipping authentication")
+		}
+	} else {
+		req.Header.Set(authHeaderName, fmt.Sprintf(authHeaderFormat, conf.Disco.Netbox.NetboxAPIToken.String()))
+	}
+
+	res, err := client.Do(req)
+	if err != nil {
+		return nil, err
+	}
+	body, err := io.ReadAll(res.Body)
+	if err != nil {
+		return nil, err
+	}
+
+	if res.StatusCode == http.StatusOK {
+		nbRes := NBRespOK{}
+		err = json.Unmarshal(body, &nbRes)
+		if err != nil {
+			return nil, err
+		}
+		return &nbRes, nil
+	} else {
+		log.Warnf("Invalid response from netbox server: %v", res.Status)
+		myErr := map[string]interface{}{}
+		err := json.Unmarshal(body, &myErr)
+		if err != nil {
+			return nil, err
+		}
+		return nil, fmt.Errorf("%v", myErr)
+	}
+}
+
+// See if any the fields in the conf are in the devices custom_fields map.
+func checkCustomFields(conf *kt.SnmpConfig, res NBResult) bool {
+	for k, v := range conf.Disco.Netbox.CustomFields {
+		if val, ok := res.CustomFields[k]; ok {
+			switch tv := val.(type) {
+			case string:
+				if v == tv {
+					return true
+				}
+			}
+		}
+	}
+
+	return false
+}
+
 func getDevicesFromNetbox(ctx context.Context, ctl chan bool, foundDevices map[string]*kt.SnmpDeviceConfig,
 	mdb *mibs.MibDB, conf *kt.SnmpConfig, kentikDevices map[string]string, log logger.ContextL, ignoreMap map[string]bool) error {
-	c := netbox.NewAPIClientFor(conf.Disco.Netbox.NetboxAPIHost, conf.Disco.Netbox.NetboxAPIToken.String())
+
+	log.Infof("Discovering devices from Netbox.")
 
 	var getDeviceList func(offset int32, results *[]scan.Result) error
 	getDeviceList = func(offset int32, results *[]scan.Result) error {
 		limit := int32(500) // @TODO, what's a good limit here?
-		res, _, err := c.DcimAPI.
-			DcimDevicesList(ctx).
-			Status([]string{"active"}). // Only look at active devices.
-			Limit(limit).
-			Offset(offset).
-			Execute()
+		res, err := getDcimDevicesApi(ctx, conf, log, offset, limit)
 		if err != nil {
 			return err
 		}
-
 		for _, res := range res.Results {
-			keep := true // Default to add all.
-
-			if conf.Disco.Netbox.NetboxTag != "" {
-				keep = false // If we are filtering with tags, only allow those tags which match though.
-				for _, tag := range res.Tags {
-					if tag.Display == conf.Disco.Netbox.NetboxTag {
-						keep = true
-						continue
-					}
-				}
-			}
-			if conf.Disco.Netbox.NetboxSite != "" { // Furthermore, if we filter with site, only the selected site.
-				if res.Site.GetDisplay() != conf.Disco.Netbox.NetboxSite {
-					keep = false
+			if len(conf.Disco.Netbox.CustomFields) > 0 && len(res.CustomFields) > 0 {
+				if !checkCustomFields(conf, res) {
+					continue
 				}
 			}
 
-			if keep {
-				ipv, err := getIP(res, conf.Disco.Netbox)
-				if err != nil {
-					log.Infof("Skipping %s with bad IP: %v", res.Display, err)
+			ipv, err := getIP(res, conf.Disco.Netbox, log)
+			if err != nil {
+				if res.Name != nil {
+					log.Warnf("Skipping %v with bad IP: %v", *res.Name, err)
+				} else {
+					log.Warnf("Skipping null device with bad IP: %v", err)
+				}
+			} else {
+				if res.Name != nil {
+					model := "unknown"
+					if res.DeviceType != nil && res.DeviceType.Model != nil {
+						model = *res.DeviceType.Model
+					}
+					*results = append(*results, scan.Result{Name: *res.Name, Manufacturer: model, Host: net.ParseIP(ipv.Addr().String())})
 				} else {
-					*results = append(*results, scan.Result{Name: res.Display, Manufacturer: res.DeviceType.GetDisplay(), Host: net.ParseIP(ipv.Addr().String())})
+					log.Warnf("Skipping device with IP %v because of null Name value.", ipv.Addr().String())
 				}
 			}
 		}
 
 		// Now, do we need to recurse further to get more results?
-		nextOffset := getNextOffset(res.Next)
+		nextOffset := getNextOffset(res.Next, log)
 		if nextOffset > offset {
 			return getDeviceList(nextOffset, results)
 		}
@@ -102,12 +322,11 @@ func getDevicesFromNetbox(ctx context.Context, ctl chan bool, foundDevices map[s
 }
 
 // Pull out the offset value from a url.
-func getNextOffset(next netbox.NullableString) int32 {
-	val := next.Get()
-	if val == nil {
+func getNextOffset(next *string, log logger.ContextL) int32 {
+	if next == nil {
 		return 0
 	}
-	u, err := url.Parse(*val)
+	u, err := url.Parse(*next)
 	if err != nil {
 		return 0
 	}
@@ -119,27 +338,30 @@ func getNextOffset(next netbox.NullableString) int32 {
 	return int32(no)
 }
 
-func getIP(res netbox.DeviceWithConfigContext, conf *kt.NetboxConfig) (netip.Prefix, error) {
+func getIP(res NBResult, conf *kt.NetboxConfig, log logger.ContextL) (netip.Prefix, error) {
 	switch conf.NetboxIP {
 	case "primary":
-		if res.PrimaryIp.IsSet() {
-			addr := res.PrimaryIp.Get().GetAddress()
+		log.Infof("Looking at primary_ip %s", res.PrimaryIp.GetVal())
+		if res.PrimaryIp != nil && res.PrimaryIp.Address != nil {
+			addr := *res.PrimaryIp.Address
 			if addr != "" {
 				ipv, err := netip.ParsePrefix(addr)
 				return ipv, err
 			}
 		}
 	case "oob":
-		if res.OobIp.IsSet() {
-			addr := res.OobIp.Get().GetAddress()
+		log.Infof("Looking at oob %v", res.OobIp.GetVal())
+		if res.OobIp != nil && res.OobIp.Address != nil {
+			addr := *res.OobIp.Address
 			if addr != "" {
 				ipv, err := netip.ParsePrefix(addr)
 				return ipv, err
 			}
 		}
 	default:
-		if res.PrimaryIp.IsSet() {
-			addr := res.PrimaryIp.Get().GetAddress()
+		log.Infof("Looking at primary_ip %v", res.PrimaryIp.GetVal())
+		if res.PrimaryIp != nil && res.PrimaryIp.Address != nil {
+			addr := *res.PrimaryIp.Address
 			if addr != "" {
 				ipv, err := netip.ParsePrefix(addr)
 				return ipv, err
diff --git a/pkg/kt/snmp.go b/pkg/kt/snmp.go
index 0ee60bb..e353934 100644
--- a/pkg/kt/snmp.go
+++ b/pkg/kt/snmp.go
@@ -268,11 +268,16 @@ type SnmpDiscoConfig struct {
 }
 
 type NetboxConfig struct {
-	NetboxAPIHost  string       `yaml:"host"`
-	NetboxAPIToken *SecureToken `yaml:"token"`
-	NetboxTag      string       `yaml:"tag"`
-	NetboxSite     string       `yaml:"site"`
-	NetboxIP       string       `yaml:"ip_to_pick"` // One of primary or oob. Default to primary.
+	NetboxAPIUrl   string            `yaml:"url"`
+	NetboxAPIToken *SecureToken      `yaml:"token"`
+	NetboxIP       string            `yaml:"ip_to_pick"` // One of primary or oob. Default to primary.
+	Tag            string            `yaml:"tag"`
+	Site           string            `yaml:"site"`
+	Location       string            `yaml:"location"`
+	Tenant         string            `yaml:"tenant"`
+	Status         string            `yaml:"status"`
+	Role           string            `yaml:"role"`
+	CustomFields   map[string]string `yaml:"custom_fields"`
 }
 
 type ProviderMap struct {


Test patch:
diff --git a/pkg/inputs/snmp/netbox_test.go b/pkg/inputs/snmp/netbox_test.go
new file mode 100644
index 0000000..b73da32
--- /dev/null
+++ b/pkg/inputs/snmp/netbox_test.go
@@ -0,0 +1,34 @@
+package snmp
+
+import (
+	"encoding/json"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+
+	"github.com/kentik/ktranslate/pkg/eggs/logger"
+	lt "github.com/kentik/ktranslate/pkg/eggs/logger/testing"
+	"github.com/kentik/ktranslate/pkg/kt"
+)
+
+func TestGetIP(t *testing.T) {
+	assert := assert.New(t)
+	l := lt.NewTestContextL(logger.NilContext, t)
+
+	test := []byte(`{
+        "id": 1639742,
+        "family": {
+          "value": 4,
+          "label": "IPv4"
+        },
+        "address": "10.249.157.132/32"
+      }`)
+
+	conf := kt.NetboxConfig{NetboxIP: "primary"}
+	nbRes := NBIP{}
+	err := json.Unmarshal(test, &nbRes)
+	assert.Nil(err)
+	ipv, err := getIP(NBResult{PrimaryIp: &nbRes}, &conf, l)
+	assert.Nil(err)
+	assert.Equal("10.249.157.132", ipv.Addr().String())
+}