Repository: coredns/coredns (go)

Issue:
Non-existent CNAME target in the same zone should be returned with NXDOMAIN instead of NOERROR rcode
Hi,

This is related to if 'A CNAME B' exists in a zone, but B (related to the same zone) does not exist, then the return code should be NXDOMAIN.  [RFC 6604](https://tools.ietf.org/html/rfc6604#section-3) mentions that 
> When chains are followed the RCODE in the ultimate DNS response MUST BE set based on the final query cycle leading to that
>       response.


Consider the following sample zone file

|  	|  	|  	|  	|
|-	|-	|-	|-	|
| example.com. 	| 500 | SOA 	| ns1.outside.com. root.example.com. 3 604800 86400 2419200 604800 	|
| example.com. 	| 500 | NS 	| ns1.outside.com.  	|
| b.c.example.com. 	| 500 | CNAME | c.c.example.com.  	|
| c.c.example.com. 	| 500 | CNAME	| d.c.example.com.  	|

For the query _<b.c.example.com. , A>_, the Coredns server returns the following response:
```
          "opcode QUERY",
          "rcode NOERROR",
          "flags QR AA",
          ";QUESTION",
          "b.c.campus.edu. IN A",
          ";ANSWER",
          "b.c.example.com. 500 IN CNAME c.c.example.com.",
          "c.c.example.com. 500 IN CNAME d.c.example.com.",
          ";AUTHORITY",
          "example.com. 500 IN NS ns1.outside.com.",
          ";ADDITIONAL"
```

**What you expected to happen**:
The expected response is the same as above except that the `rcode` should be `NXDOMAIN`. 

**Environment**:

- the version of CoreDNS: `CoreDNS-1.8.0  linux/amd64, go1.15.3, 054c9ae`
- Corefile:
```
example.com:53 {
    file db.example.com
    log
    errors
}
```
- OS: Ubuntu 18


Pull request:
plugin/file: Use NXDOMAIN response if CNAME target is NXDOMAIN
Signed-off-by: Chris O'Haver <cohaver@infoblox.com>

### 1. Why is this pull request needed and what does it do?

As cited in  #4288, per https://tools.ietf.org/html/rfc6604#section-3, when following a CNAME chain the response code sent to the client should be the same response code received when resolving the target of the final link in the chain.  Strictly this means we should be passing through the exact Rcode received when looking up the final target. But since _file_ uses an alternate response code datatype `file.Result` which does not map 1:1 to DNS response codes, making that happen might require some larger scope refactoring.  So for now, just addressing response codes that do map.

### 2. Which issues (if any) are related?

fixes #4288

### 3. Which documentation changes (if any) need to be made?

### 4. Does this introduce a backward incompatible change or deprecation?


Patch:
diff --git a/go.mod b/go.mod
index 1c4678e6b..4455c2924 100644
--- a/go.mod
+++ b/go.mod
@@ -4,14 +4,14 @@ go 1.13
 
 require (
 	github.com/Azure/azure-sdk-for-go v40.6.0+incompatible
-	github.com/Azure/go-autorest/autorest v0.11.11
-	github.com/Azure/go-autorest/autorest/azure/auth v0.5.3
+	github.com/Azure/go-autorest/autorest v0.11.13
+	github.com/Azure/go-autorest/autorest/azure/auth v0.5.5
 	github.com/Azure/go-autorest/autorest/to v0.2.0 // indirect
 	github.com/DataDog/datadog-go v3.5.0+incompatible // indirect
-	github.com/aws/aws-sdk-go v1.35.28
+	github.com/aws/aws-sdk-go v1.36.2
 	github.com/cenkalti/backoff/v4 v4.1.0
 	github.com/coredns/caddy v1.1.0
-	github.com/dnstap/golang-dnstap v0.2.2
+	github.com/dnstap/golang-dnstap v0.3.0
 	github.com/dustin/go-humanize v1.0.0 // indirect
 	github.com/farsightsec/golang-framestream v0.3.0
 	github.com/golang/protobuf v1.4.3
diff --git a/go.sum b/go.sum
index 0b3f70cbb..821d843d4 100644
--- a/go.sum
+++ b/go.sum
@@ -29,15 +29,15 @@ github.com/Azure/go-autorest v14.2.0+incompatible h1:V5VMDjClD3GiElqLWO7mz2MxNAK
 github.com/Azure/go-autorest v14.2.0+incompatible/go.mod h1:r+4oMnoxhatjLLJ6zxSWATqVooLgysK6ZNox3g/xq24=
 github.com/Azure/go-autorest/autorest v0.9.0/go.mod h1:xyHB1BMZT0cuDHU7I0+g046+BFDTQ8rEZB0s4Yfa6bI=
 github.com/Azure/go-autorest/autorest v0.9.6/go.mod h1:/FALq9T/kS7b5J5qsQ+RSTUdAmGFqi0vUdVNNx8q630=
-github.com/Azure/go-autorest/autorest v0.11.9/go.mod h1:eipySxLmqSyC5s5k1CLupqet0PSENBEDP93LQ9a8QYw=
-github.com/Azure/go-autorest/autorest v0.11.11 h1:k/wzH9pA3hrtFNsEhJ5SqPEs75W3bzS8VOYA/fJ0j1k=
-github.com/Azure/go-autorest/autorest v0.11.11/go.mod h1:eipySxLmqSyC5s5k1CLupqet0PSENBEDP93LQ9a8QYw=
+github.com/Azure/go-autorest/autorest v0.11.13 h1:XKx/sB3bfadpXBBHPc7tP2XPKhzVyrdhxpDC3T0wqjs=
+github.com/Azure/go-autorest/autorest v0.11.13/go.mod h1:eipySxLmqSyC5s5k1CLupqet0PSENBEDP93LQ9a8QYw=
 github.com/Azure/go-autorest/autorest/adal v0.5.0/go.mod h1:8Z9fGy2MpX0PvDjB1pEgQTmVqjGhiHBW7RJJEciWzS0=
 github.com/Azure/go-autorest/autorest/adal v0.8.2/go.mod h1:ZjhuQClTqx435SRJ2iMlOxPYt3d2C/T/7TiQCVZSn3Q=
-github.com/Azure/go-autorest/autorest/adal v0.9.5 h1:Y3bBUV4rTuxenJJs41HU3qmqsb+auo+a3Lz+PlJPpL0=
 github.com/Azure/go-autorest/autorest/adal v0.9.5/go.mod h1:B7KF7jKIeC9Mct5spmyCB/A8CG/sEz1vwIRGv/bbw7A=
-github.com/Azure/go-autorest/autorest/azure/auth v0.5.3 h1:lZifaPRAk1bqg5vGqreL6F8uLC5V0fDpY8nFvc3boFc=
-github.com/Azure/go-autorest/autorest/azure/auth v0.5.3/go.mod h1:4bJZhUhcq8LB20TruwHbAQsmUs2Xh+QR7utuJpLXX3A=
+github.com/Azure/go-autorest/autorest/adal v0.9.8 h1:bW6ZdxqMYWsxGikpM62SSE3jnvOXVu9SXzJTuj1WM3Y=
+github.com/Azure/go-autorest/autorest/adal v0.9.8/go.mod h1:B7KF7jKIeC9Mct5spmyCB/A8CG/sEz1vwIRGv/bbw7A=
+github.com/Azure/go-autorest/autorest/azure/auth v0.5.5 h1:7HT2JTm2BOsBMPrT1/iWZW4+XmRvyICcbCejf9BkmYU=
+github.com/Azure/go-autorest/autorest/azure/auth v0.5.5/go.mod h1:ptW4D47I+eIUe/lulFLYTVfG4rAARZoXIe1vmTQ+ol8=
 github.com/Azure/go-autorest/autorest/azure/cli v0.4.2 h1:dMOmEJfkLKW/7JsokJqkyoYSgmR08hi9KrhjZb+JALY=
 github.com/Azure/go-autorest/autorest/azure/cli v0.4.2/go.mod h1:7qkJkT+j6b+hIpzMOwPChJhTqS8VbsqqgULzMNRugoM=
 github.com/Azure/go-autorest/autorest/date v0.1.0/go.mod h1:plvfp3oPSKwf2DNjlBjWF/7vwR+cUD/ELuzDCXwHUVA=
@@ -83,8 +83,8 @@ github.com/armon/go-radix v0.0.0-20180808171621-7fddfc383310/go.mod h1:ufUuZ+zHj
 github.com/aryann/difflib v0.0.0-20170710044230-e206f873d14a/go.mod h1:DAHtR1m6lCRdSC2Tm3DSWRPvIPr6xNKyeHdqDQSQT+A=
 github.com/aws/aws-lambda-go v1.13.3/go.mod h1:4UKl9IzQMoD+QF79YdCuzCwp8VbmG4VAQwij/eHl5CU=
 github.com/aws/aws-sdk-go v1.27.0/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
-github.com/aws/aws-sdk-go v1.35.28 h1:S2LuRnfC8X05zgZLC8gy/Sb82TGv2Cpytzbzz7tkeHc=
-github.com/aws/aws-sdk-go v1.35.28/go.mod h1:tlPOdRjfxPBpNIwqDj61rmsnA85v9jc0Ps9+muhnW+k=
+github.com/aws/aws-sdk-go v1.36.2 h1:UAeFPct+jHqWM+tgiqDrC9/sfbWj6wkcvpsJ+zdcsvA=
+github.com/aws/aws-sdk-go v1.36.2/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro=
 github.com/aws/aws-sdk-go-v2 v0.18.0/go.mod h1:JWVYvqSMppoMJC0x5wdwiImzgXTI9FuZwxzkQq9wy+g=
 github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
 github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
@@ -127,8 +127,8 @@ github.com/dgrijalva/jwt-go v3.2.0+incompatible h1:7qlOGliEKZXTDg6OTjfoBKDXWrumC
 github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
 github.com/dimchansky/utfbom v1.1.0 h1:FcM3g+nofKgUteL8dm/UpdRXNC9KmADgTpLKsu0TRo4=
 github.com/dimchansky/utfbom v1.1.0/go.mod h1:rO41eb7gLfo8SF1jd9F8HplJm1Fewwi4mQvIirEdv+8=
-github.com/dnstap/golang-dnstap v0.2.2 h1:H2PCf5R92SRSsXuSM9P4ToqRkzaRLwtS8UPwauHLPf8=
-github.com/dnstap/golang-dnstap v0.2.2/go.mod h1:JIH+8jjV4pSEZCGPfPgFfuwyzmAuTLrEnk0BxtrF/5w=
+github.com/dnstap/golang-dnstap v0.3.0 h1:eAgWqfNFlPVlplDQFpj1V9qtoP3On5ZVX2Vl/+hHZx4=
+github.com/dnstap/golang-dnstap v0.3.0/go.mod h1:49IwPRTxiC2Jz0jwrLqiPnRyevwKpAQ/tmO0P0hZO5o=
 github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96/go.mod h1:Qh8CwZgvJUkLughtfhJv5dyTYa91l1fOUCrgjqmcifM=
 github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE=
 github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
@@ -147,7 +147,6 @@ github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1m
 github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
 github.com/evanphx/json-patch v4.9.0+incompatible h1:kLcOMZeuLAJvL2BPWLMIj5oaZQobrkAqrL+WFZwQses=
 github.com/evanphx/json-patch v4.9.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
-github.com/farsightsec/golang-framestream v0.2.0/go.mod h1:eNde4IQyEiA5br02AouhEHCu3p3UzrCdFR4LuQHklMI=
 github.com/farsightsec/golang-framestream v0.3.0 h1:/spFQHucTle/ZIPkYqrfshQqPe2VQEzesH243TjIwqA=
 github.com/farsightsec/golang-framestream v0.3.0/go.mod h1:eNde4IQyEiA5br02AouhEHCu3p3UzrCdFR4LuQHklMI=
 github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
@@ -568,8 +567,9 @@ golang.org/x/net v0.0.0-20200222125558-5a598a2470a0/go.mod h1:z5CRVTTTmAJ677TzLL
 golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
 golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
-golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU=
 golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
+golang.org/x/net v0.0.0-20201110031124-69a78807bb2b h1:uwuIcX0g4Yl1NC5XAz37xsr2lTtcqevgzYNVt49waME=
+golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -619,6 +619,7 @@ golang.org/x/sys v0.0.0-20200331124033-c3d80250170d/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200622214017-ed371f2e16b4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20201015000850-e3ed0017c211 h1:9UQO31fZ+0aKQOFldThf7BKPMJTiBfWycGh/u3UoO88=
 golang.org/x/sys v0.0.0-20201015000850-e3ed0017c211/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
diff --git a/plugin/file/lookup.go b/plugin/file/lookup.go
index 28da3d4d0..e036d809a 100644
--- a/plugin/file/lookup.go
+++ b/plugin/file/lookup.go
@@ -307,8 +307,9 @@ func (z *Zone) externalLookup(ctx context.Context, state request.Request, elem *
 	targetName := rrs[0].(*dns.CNAME).Target
 	elem, _ = z.Tree.Search(targetName)
 	if elem == nil {
-		rrs = append(rrs, z.doLookup(ctx, state, targetName, qtype)...)
-		return rrs, z.Apex.ns(do), nil, Success
+		lookupRRs, result := z.doLookup(ctx, state, targetName, qtype)
+		rrs = append(rrs, lookupRRs...)
+		return rrs, z.Apex.ns(do), nil, result
 	}
 
 	i := 0
@@ -326,8 +327,9 @@ Redo:
 		targetName := cname[0].(*dns.CNAME).Target
 		elem, _ = z.Tree.Search(targetName)
 		if elem == nil {
-			rrs = append(rrs, z.doLookup(ctx, state, targetName, qtype)...)
-			return rrs, z.Apex.ns(do), nil, Success
+			lookupRRs, result := z.doLookup(ctx, state, targetName, qtype)
+			rrs = append(rrs, lookupRRs...)
+			return rrs, z.Apex.ns(do), nil, result
 		}
 
 		i++
@@ -352,15 +354,24 @@ Redo:
 	return rrs, z.Apex.ns(do), nil, Success
 }
 
-func (z *Zone) doLookup(ctx context.Context, state request.Request, target string, qtype uint16) []dns.RR {
+func (z *Zone) doLookup(ctx context.Context, state request.Request, target string, qtype uint16) ([]dns.RR, Result) {
 	m, e := z.Upstream.Lookup(ctx, state, target, qtype)
 	if e != nil {
-		return nil
+		return nil, Success
 	}
 	if m == nil {
-		return nil
+		return nil, Success
+	}
+	if m.Rcode == dns.RcodeNameError {
+		return m.Answer, NameError
+	}
+	if m.Rcode == dns.RcodeServerFailure {
+		return m.Answer, ServerFailure
+	}
+	if m.Rcode == dns.RcodeSuccess && len(m.Answer) == 0 {
+		return m.Answer, NoData
 	}
-	return m.Answer
+	return m.Answer, Success
 }
 
 // additionalProcessing checks the current answer section and retrieves A or AAAA records
diff --git a/plugin/kubernetes/controller.go b/plugin/kubernetes/controller.go
index f9373eee9..2319cf203 100644
--- a/plugin/kubernetes/controller.go
+++ b/plugin/kubernetes/controller.go
@@ -92,9 +92,8 @@ type dnsControlOpts struct {
 	namespaceLabelSelector *meta.LabelSelector
 	namespaceSelector      labels.Selector
 
-	zones                 []string
-	endpointNameMode      bool
-	skipAPIObjectsCleanup bool
+	zones            []string
+	endpointNameMode bool
 }
 
 // newDNSController creates a controller for CoreDNS.
@@ -116,7 +115,7 @@ func newdnsController(ctx context.Context, kubeClient kubernetes.Interface, opts
 		&api.Service{},
 		cache.ResourceEventHandlerFuncs{AddFunc: dns.Add, UpdateFunc: dns.Update, DeleteFunc: dns.Delete},
 		cache.Indexers{svcNameNamespaceIndex: svcNameNamespaceIndexFunc, svcIPIndex: svcIPIndexFunc},
-		object.DefaultProcessor(object.ToService(opts.skipAPIObjectsCleanup), nil),
+		object.DefaultProcessor(object.ToService, nil),
 	)
 
 	if opts.initPodCache {
@@ -128,7 +127,7 @@ func newdnsController(ctx context.Context, kubeClient kubernetes.Interface, opts
 			&api.Pod{},
 			cache.ResourceEventHandlerFuncs{AddFunc: dns.Add, UpdateFunc: dns.Update, DeleteFunc: dns.Delete},
 			cache.Indexers{podIPIndex: podIPIndexFunc},
-			object.DefaultProcessor(object.ToPod(opts.skipAPIObjectsCleanup), nil),
+			object.DefaultProcessor(object.ToPod, nil),
 		)
 	}
 
@@ -136,28 +135,28 @@ func newdnsController(ctx context.Context, kubeClient kubernetes.Interface, opts
 		var (
 			apiObj    runtime.Object
 			listWatch cache.ListWatch
-			to        func(bool) object.ToFunc
-			latency   object.RecordLatencyFunc
+			to        object.ToFunc
+			latency   *object.EndpointLatencyRecorder
 		)
 		if opts.useEndpointSlices {
 			apiObj = &discovery.EndpointSlice{}
 			listWatch.ListFunc = endpointSliceListFunc(ctx, dns.client, api.NamespaceAll, dns.selector)
 			listWatch.WatchFunc = endpointSliceWatchFunc(ctx, dns.client, api.NamespaceAll, dns.selector)
 			to = object.EndpointSliceToEndpoints
-			latency = dns.recordEndpointSliceDNSProgrammingLatency
+			latency = dns.EndpointSliceLatencyRecorder()
 		} else {
 			apiObj = &api.Endpoints{}
 			listWatch.ListFunc = endpointsListFunc(ctx, dns.client, api.NamespaceAll, dns.selector)
 			listWatch.WatchFunc = endpointsWatchFunc(ctx, dns.client, api.NamespaceAll, dns.selector)
 			to = object.ToEndpoints
-			latency = dns.recordEndpointDNSProgrammingLatency
+			latency = dns.EndpointsLatencyRecorder()
 		}
 		dns.epLister, dns.epController = object.NewIndexerInformer(
 			&listWatch,
 			apiObj,
 			cache.ResourceEventHandlerFuncs{AddFunc: dns.Add, UpdateFunc: dns.Update, DeleteFunc: dns.Delete},
 			cache.Indexers{epNameNamespaceIndex: epNameNamespaceIndexFunc, epIPIndex: epIPIndexFunc},
-			object.DefaultProcessor(to(opts.skipAPIObjectsCleanup), latency),
+			object.DefaultProcessor(to, latency),
 		)
 	}
 
@@ -173,12 +172,19 @@ func newdnsController(ctx context.Context, kubeClient kubernetes.Interface, opts
 	return &dns
 }
 
-func (dns *dnsControl) recordEndpointDNSProgrammingLatency(obj meta.Object) {
-	recordDNSProgrammingLatency(dns.getServices(obj.(*api.Endpoints)), obj)
+func (dns *dnsControl) EndpointsLatencyRecorder() *object.EndpointLatencyRecorder {
+	return &object.EndpointLatencyRecorder{
+		ServiceFunc: func(o meta.Object) []*object.Service {
+			return dns.SvcIndex(object.ServiceKey(o.GetName(), o.GetNamespace()))
+		},
+	}
 }
-
-func (dns *dnsControl) recordEndpointSliceDNSProgrammingLatency(obj meta.Object) {
-	recordDNSProgrammingLatency(dns.SvcIndex(object.ServiceKey(obj.GetLabels()[discovery.LabelServiceName], obj.GetNamespace())), obj)
+func (dns *dnsControl) EndpointSliceLatencyRecorder() *object.EndpointLatencyRecorder {
+	return &object.EndpointLatencyRecorder{
+		ServiceFunc: func(o meta.Object) []*object.Service {
+			return dns.SvcIndex(object.ServiceKey(o.GetLabels()[discovery.LabelServiceName], o.GetNamespace()))
+		},
+	}
 }
 
 func podIPIndexFunc(obj interface{}) ([]string, error) {
@@ -518,10 +524,6 @@ func (dns *dnsControl) detectChanges(oldObj, newObj interface{}) {
 	}
 }
 
-func (dns *dnsControl) getServices(endpoints *api.Endpoints) []*object.Service {
-	return dns.SvcIndex(object.ServiceKey(endpoints.GetName(), endpoints.GetNamespace()))
-}
-
 // subsetsEquivalent checks if two endpoint subsets are significantly equivalent
 // I.e. that they have the same ready addresses, host names, ports (including protocol
 // and service names for SRV)
diff --git a/plugin/kubernetes/object/endpoint.go b/plugin/kubernetes/object/endpoint.go
index 304aaa861..09429e0b2 100644
--- a/plugin/kubernetes/object/endpoint.go
+++ b/plugin/kubernetes/object/endpoint.go
@@ -5,6 +5,7 @@ import (
 
 	api "k8s.io/api/core/v1"
 	discovery "k8s.io/api/discovery/v1beta1"
+	meta "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/runtime"
 )
 
@@ -46,30 +47,12 @@ type EndpointPort struct {
 // EndpointsKey returns a string using for the index.
 func EndpointsKey(name, namespace string) string { return name + "." + namespace }
 
-// ToEndpoints returns a function that converts an *api.Endpoints to a *Endpoints.
-func ToEndpoints(skipCleanup bool) ToFunc {
-	return func(obj interface{}) (interface{}, error) {
-		eps, ok := obj.(*api.Endpoints)
-		if !ok {
-			return nil, fmt.Errorf("unexpected object %v", obj)
-		}
-		return toEndpoints(skipCleanup, eps), nil
-	}
-}
-
-// EndpointSliceToEndpoints returns a function that converts an *discovery.EndpointSlice to a *Endpoints.
-func EndpointSliceToEndpoints(skipCleanup bool) ToFunc {
-	return func(obj interface{}) (interface{}, error) {
-		eps, ok := obj.(*discovery.EndpointSlice)
-		if !ok {
-			return nil, fmt.Errorf("unexpected object %v", obj)
-		}
-		return endpointSliceToEndpoints(skipCleanup, eps), nil
-	}
-}
-
 // toEndpoints converts an *api.Endpoints to a *Endpoints.
-func toEndpoints(skipCleanup bool, end *api.Endpoints) *Endpoints {
+func ToEndpoints(obj meta.Object) (meta.Object, error) {
+	end, ok := obj.(*api.Endpoints)
+	if !ok {
+		return nil, fmt.Errorf("unexpected object %v", obj)
+	}
 	e := &Endpoints{
 		Version:   end.GetResourceVersion(),
 		Name:      end.GetName(),
@@ -113,15 +96,17 @@ func toEndpoints(skipCleanup bool, end *api.Endpoints) *Endpoints {
 		}
 	}
 
-	if !skipCleanup {
-		*end = api.Endpoints{}
-	}
+	*end = api.Endpoints{}
 
-	return e
+	return e, nil
 }
 
-// endpointSliceToEndpoints converts a *discovery.EndpointSlice to a *Endpoints.
-func endpointSliceToEndpoints(skipCleanup bool, ends *discovery.EndpointSlice) *Endpoints {
+// EndpointSliceToEndpoints converts a *discovery.EndpointSlice to a *Endpoints.
+func EndpointSliceToEndpoints(obj meta.Object) (meta.Object, error) {
+	ends, ok := obj.(*discovery.EndpointSlice)
+	if !ok {
+		return nil, fmt.Errorf("unexpected object %v", obj)
+	}
 	e := &Endpoints{
 		Version:   ends.GetResourceVersion(),
 		Name:      ends.GetName(),
@@ -156,11 +141,9 @@ func endpointSliceToEndpoints(skipCleanup bool, ends *discovery.EndpointSlice) *
 		}
 	}
 
-	if !skipCleanup {
-		*ends = discovery.EndpointSlice{}
-	}
+	*ends = discovery.EndpointSlice{}
 
-	return e
+	return e, nil
 }
 
 // CopyWithoutSubsets copies e, without the subsets.
diff --git a/plugin/kubernetes/object/informer.go b/plugin/kubernetes/object/informer.go
index afd134e56..e44cd218a 100644
--- a/plugin/kubernetes/object/informer.go
+++ b/plugin/kubernetes/object/informer.go
@@ -1,6 +1,8 @@
 package object
 
 import (
+	"fmt"
+
 	meta "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/client-go/tools/cache"
@@ -25,13 +27,18 @@ func NewIndexerInformer(lw cache.ListerWatcher, objType runtime.Object, h cache.
 type RecordLatencyFunc func(meta.Object)
 
 // DefaultProcessor is based on the Process function from cache.NewIndexerInformer except it does a conversion.
-func DefaultProcessor(convert ToFunc, recordLatency RecordLatencyFunc) ProcessorBuilder {
+func DefaultProcessor(convert ToFunc, recordLatency *EndpointLatencyRecorder) ProcessorBuilder {
 	return func(clientState cache.Indexer, h cache.ResourceEventHandler) cache.ProcessFunc {
 		return func(obj interface{}) error {
 			for _, d := range obj.(cache.Deltas) {
+				if recordLatency != nil {
+					if o, ok := d.Object.(meta.Object); ok {
+						recordLatency.init(o)
+					}
+				}
 				switch d.Type {
 				case cache.Sync, cache.Added, cache.Updated:
-					obj, err := convert(d.Object)
+					obj, err := convert(d.Object.(meta.Object))
 					if err != nil {
 						return err
 					}
@@ -47,14 +54,18 @@ func DefaultProcessor(convert ToFunc, recordLatency RecordLatencyFunc) Processor
 						h.OnAdd(obj)
 					}
 					if recordLatency != nil {
-						recordLatency(d.Object.(meta.Object))
+						recordLatency.record()
 					}
 				case cache.Deleted:
 					var obj interface{}
 					obj, ok := d.Object.(cache.DeletedFinalStateUnknown)
 					if !ok {
 						var err error
-						obj, err = convert(d.Object)
+						metaObj, ok := d.Object.(meta.Object)
+						if !ok {
+							return fmt.Errorf("unexpected object %v", d.Object)
+						}
+						obj, err = convert(metaObj)
 						if err != nil && err != errPodTerminating {
 							return err
 						}
@@ -65,7 +76,7 @@ func DefaultProcessor(convert ToFunc, recordLatency RecordLatencyFunc) Processor
 					}
 					h.OnDelete(obj)
 					if !ok && recordLatency != nil {
-						recordLatency(d.Object.(meta.Object))
+						recordLatency.record()
 					}
 				}
 			}
diff --git a/plugin/kubernetes/metrics.go b/plugin/kubernetes/object/metrics.go
similarity index 70%
rename from plugin/kubernetes/metrics.go
rename to plugin/kubernetes/object/metrics.go
index 8adeb6940..929925cf1 100644
--- a/plugin/kubernetes/metrics.go
+++ b/plugin/kubernetes/object/metrics.go
@@ -1,10 +1,11 @@
-package kubernetes
+package object
 
 import (
 	"time"
 
 	"github.com/coredns/coredns/plugin"
-	"github.com/coredns/coredns/plugin/kubernetes/object"
+	"github.com/coredns/coredns/plugin/pkg/log"
+
 	"github.com/prometheus/client_golang/prometheus"
 	"github.com/prometheus/client_golang/prometheus/promauto"
 	api "k8s.io/api/core/v1"
@@ -25,42 +26,50 @@ var (
 	//   * headless_without_selector
 	DNSProgrammingLatency = promauto.NewHistogramVec(prometheus.HistogramOpts{
 		Namespace: plugin.Namespace,
-		Subsystem: pluginName,
+		Subsystem: "kubernetes",
 		Name:      "dns_programming_duration_seconds",
 		// From 1 millisecond to ~17 minutes.
 		Buckets: prometheus.ExponentialBuckets(0.001, 2, 20),
 		Help:    "Histogram of the time (in seconds) it took to program a dns instance.",
 	}, []string{"service_kind"})
 
-	// durationSinceFunc returns the duration elapsed since the given time.
+	// DurationSinceFunc returns the duration elapsed since the given time.
 	// Added as a global variable to allow injection for testing.
-	durationSinceFunc = time.Since
+	DurationSinceFunc = time.Since
 )
 
-func recordDNSProgrammingLatency(svcs []*object.Service, endpoints meta.Object) {
-	// getLastChangeTriggerTime is the time.Time value of the EndpointsLastChangeTriggerTime
-	// annotation stored in the given endpoints object or the "zero" time if the annotation wasn't set
-	var lastChangeTriggerTime time.Time
-	stringVal, ok := endpoints.GetAnnotations()[api.EndpointsLastChangeTriggerTime]
+// EndpointLatencyRecorder records latency metric for endpoint objects
+type EndpointLatencyRecorder struct {
+	TT          time.Time
+	ServiceFunc func(meta.Object) []*Service
+	Services    []*Service
+}
+
+func (l *EndpointLatencyRecorder) init(o meta.Object) {
+	l.Services = l.ServiceFunc(o)
+	l.TT = time.Time{}
+	stringVal, ok := o.GetAnnotations()[api.EndpointsLastChangeTriggerTime]
 	if ok {
-		ts, err := time.Parse(time.RFC3339Nano, stringVal)
+		tt, err := time.Parse(time.RFC3339Nano, stringVal)
 		if err != nil {
 			log.Warningf("DnsProgrammingLatency cannot be calculated for Endpoints '%s/%s'; invalid %q annotation RFC3339 value of %q",
-				endpoints.GetNamespace(), endpoints.GetName(), api.EndpointsLastChangeTriggerTime, stringVal)
-			// In case of error val = time.Zero, which is ignored in the upstream code.
+				o.GetNamespace(), o.GetName(), api.EndpointsLastChangeTriggerTime, stringVal)
+			// In case of error val = time.Zero, which is ignored downstream.
 		}
-		lastChangeTriggerTime = ts
+		l.TT = tt
 	}
+}
 
+func (l *EndpointLatencyRecorder) record() {
 	// isHeadless indicates whether the endpoints object belongs to a headless
 	// service (i.e. clusterIp = None). Note that this can be a  false negatives if the service
 	// informer is lagging, i.e. we may not see a recently created service. Given that the services
 	// don't change very often (comparing to much more frequent endpoints changes), cases when this method
 	// will return wrong answer should be relatively rare. Because of that we intentionally accept this
 	// flaw to keep the solution simple.
-	isHeadless := len(svcs) == 1 && svcs[0].ClusterIP == api.ClusterIPNone
+	isHeadless := len(l.Services) == 1 && l.Services[0].ClusterIP == api.ClusterIPNone
 
-	if endpoints == nil || !isHeadless || lastChangeTriggerTime.IsZero() {
+	if !isHeadless || l.TT.IsZero() {
 		return
 	}
 
@@ -69,5 +78,5 @@ func recordDNSProgrammingLatency(svcs []*object.Service, endpoints meta.Object)
 	// LastChangeTriggerTime annotation is set). It means that the corresponding service is a
 	// "headless service with selector".
 	DNSProgrammingLatency.WithLabelValues("headless_with_selector").
-		Observe(durationSinceFunc(lastChangeTriggerTime).Seconds())
+		Observe(DurationSinceFunc(l.TT).Seconds())
 }
diff --git a/plugin/kubernetes/object/object.go b/plugin/kubernetes/object/object.go
index f591f6d0a..7111833e6 100644
--- a/plugin/kubernetes/object/object.go
+++ b/plugin/kubernetes/object/object.go
@@ -22,8 +22,8 @@ import (
 	"k8s.io/client-go/tools/cache"
 )
 
-// ToFunc converts one empty interface to another.
-type ToFunc func(interface{}) (interface{}, error)
+// ToFunc converts one v1.Object to another v1.Object.
+type ToFunc func(v1.Object) (v1.Object, error)
 
 // ProcessorBuilder returns function to process cache events.
 type ProcessorBuilder func(cache.Indexer, cache.ResourceEventHandler) cache.ProcessFunc
diff --git a/plugin/kubernetes/object/pod.go b/plugin/kubernetes/object/pod.go
index 04cbe1ad2..9b9d5641c 100644
--- a/plugin/kubernetes/object/pod.go
+++ b/plugin/kubernetes/object/pod.go
@@ -5,6 +5,7 @@ import (
 	"fmt"
 
 	api "k8s.io/api/core/v1"
+	meta "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/runtime"
 )
 
@@ -21,37 +22,28 @@ type Pod struct {
 
 var errPodTerminating = errors.New("pod terminating")
 
-// ToPod returns a function that converts an api.Pod to a *Pod.
-func ToPod(skipCleanup bool) ToFunc {
-	return func(obj interface{}) (interface{}, error) {
-		apiPod, ok := obj.(*api.Pod)
-		if !ok {
-			return nil, fmt.Errorf("unexpected object %v", obj)
-		}
-		pod := toPod(skipCleanup, apiPod)
-		t := apiPod.ObjectMeta.DeletionTimestamp
-		if t != nil && !(*t).Time.IsZero() {
-			// if the pod is in the process of termination, return an error so it can be ignored
-			// during add/update event processing
-			return pod, errPodTerminating
-		}
-		return pod, nil
+// ToPod converts an api.Pod to a *Pod.
+func ToPod(obj meta.Object) (meta.Object, error) {
+	apiPod, ok := obj.(*api.Pod)
+	if !ok {
+		return nil, fmt.Errorf("unexpected object %v", obj)
 	}
-}
-
-func toPod(skipCleanup bool, pod *api.Pod) *Pod {
-	p := &Pod{
-		Version:   pod.GetResourceVersion(),
-		PodIP:     pod.Status.PodIP,
-		Namespace: pod.GetNamespace(),
-		Name:      pod.GetName(),
+	pod := &Pod{
+		Version:   apiPod.GetResourceVersion(),
+		PodIP:     apiPod.Status.PodIP,
+		Namespace: apiPod.GetNamespace(),
+		Name:      apiPod.GetName(),
 	}
-
-	if !skipCleanup {
-		*pod = api.Pod{}
+	t := apiPod.ObjectMeta.DeletionTimestamp
+	if t != nil && !(*t).Time.IsZero() {
+		// if the pod is in the process of termination, return an error so it can be ignored
+		// during add/update event processing
+		return pod, errPodTerminating
 	}
 
-	return p
+	*apiPod = api.Pod{}
+
+	return pod, nil
 }
 
 var _ runtime.Object = &Pod{}
diff --git a/plugin/kubernetes/object/service.go b/plugin/kubernetes/object/service.go
index de84cf941..be1404ea0 100644
--- a/plugin/kubernetes/object/service.go
+++ b/plugin/kubernetes/object/service.go
@@ -4,6 +4,7 @@ import (
 	"fmt"
 
 	api "k8s.io/api/core/v1"
+	meta "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/runtime"
 )
 
@@ -28,18 +29,12 @@ type Service struct {
 // ServiceKey returns a string using for the index.
 func ServiceKey(name, namespace string) string { return name + "." + namespace }
 
-// ToService returns a function that converts an api.Service to a *Service.
-func ToService(skipCleanup bool) ToFunc {
-	return func(obj interface{}) (interface{}, error) {
-		svc, ok := obj.(*api.Service)
-		if !ok {
-			return nil, fmt.Errorf("unexpected object %v", obj)
-		}
-		return toService(skipCleanup, svc), nil
+// ToService converts an api.Service to a *Service.
+func ToService(obj meta.Object) (meta.Object, error) {
+	svc, ok := obj.(*api.Service)
+	if !ok {
+		return nil, fmt.Errorf("unexpected object %v", obj)
 	}
-}
-
-func toService(skipCleanup bool, svc *api.Service) *Service {
 	s := &Service{
 		Version:      svc.GetResourceVersion(),
 		Name:         svc.GetName(),
@@ -70,11 +65,9 @@ func toService(skipCleanup bool, svc *api.Service) *Service {
 
 	}
 
-	if !skipCleanup {
-		*svc = api.Service{}
-	}
+	*svc = api.Service{}
 
-	return s
+	return s, nil
 }
 
 var _ runtime.Object = &Service{}


Test patch:
diff --git a/plugin/kubernetes/informer_test.go b/plugin/kubernetes/informer_test.go
index 5156554e9..7aa9d1e83 100644
--- a/plugin/kubernetes/informer_test.go
+++ b/plugin/kubernetes/informer_test.go
@@ -11,7 +11,7 @@ import (
 )
 
 func TestDefaultProcessor(t *testing.T) {
-	pbuild := object.DefaultProcessor(object.ToService(true), nil)
+	pbuild := object.DefaultProcessor(object.ToService, nil)
 	reh := cache.ResourceEventHandlerFuncs{}
 	idx := cache.NewIndexer(cache.DeletionHandlingMetaNamespaceKeyFunc, cache.Indexers{})
 	processor := pbuild(idx, reh)
@@ -30,8 +30,8 @@ func testProcessor(t *testing.T, processor cache.ProcessFunc, idx cache.Indexer)
 
 	// Add the objects
 	err := processor(cache.Deltas{
-		{Type: cache.Added, Object: obj},
-		{Type: cache.Added, Object: obj2},
+		{Type: cache.Added, Object: obj.DeepCopy()},
+		{Type: cache.Added, Object: obj2.DeepCopy()},
 	})
 	if err != nil {
 		t.Fatalf("add failed: %v", err)
@@ -55,7 +55,7 @@ func testProcessor(t *testing.T, processor cache.ProcessFunc, idx cache.Indexer)
 	obj.Spec.ClusterIP = "1.2.3.5"
 	err = processor(cache.Deltas{{
 		Type:   cache.Updated,
-		Object: obj,
+		Object: obj.DeepCopy(),
 	}})
 	if err != nil {
 		t.Fatalf("update failed: %v", err)
@@ -78,7 +78,7 @@ func testProcessor(t *testing.T, processor cache.ProcessFunc, idx cache.Indexer)
 	// Delete an object
 	err = processor(cache.Deltas{{
 		Type:   cache.Deleted,
-		Object: obj2,
+		Object: obj2.DeepCopy(),
 	}})
 	if err != nil {
 		t.Fatalf("delete test failed: %v", err)
diff --git a/plugin/kubernetes/metrics_test.go b/plugin/kubernetes/metrics_test.backup
similarity index 58%
rename from plugin/kubernetes/metrics_test.go
rename to plugin/kubernetes/metrics_test.backup
index 43b5ca382..8274eef13 100644
--- a/plugin/kubernetes/metrics_test.go
+++ b/plugin/kubernetes/metrics_test.backup
@@ -1,20 +1,16 @@
 package kubernetes
 
 import (
-	"context"
 	"strings"
 	"testing"
 	"time"
 
 	"github.com/coredns/coredns/plugin/kubernetes/object"
-
 	"github.com/prometheus/client_golang/prometheus/testutil"
 	api "k8s.io/api/core/v1"
 	discovery "k8s.io/api/discovery/v1beta1"
 	meta "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/util/wait"
-	"k8s.io/client-go/kubernetes"
-	"k8s.io/client-go/kubernetes/fake"
+	"k8s.io/client-go/tools/cache"
 )
 
 const (
@@ -50,21 +46,19 @@ var expected = `
 	`
 
 func TestDNSProgrammingLatencyEndpointSlices(t *testing.T) {
-	client := fake.NewSimpleClientset()
 	now := time.Now()
-	ctx := context.TODO()
-	controller := newdnsController(ctx, client, dnsControlOpts{
-		initEndpointsCache: true,
-		useEndpointSlices:  true,
-		// This is needed as otherwise the fake k8s client doesn't work properly.
-		skipAPIObjectsCleanup: true,
-	})
-
-	durationSinceFunc = func(t time.Time) time.Duration {
+
+	svcIdx := cache.NewIndexer(cache.DeletionHandlingMetaNamespaceKeyFunc, cache.Indexers{svcNameNamespaceIndex: svcNameNamespaceIndexFunc})
+	epIdx := cache.NewIndexer(cache.DeletionHandlingMetaNamespaceKeyFunc, cache.Indexers{})
+
+	dns := dnsControl{svcLister: svcIdx}
+	svcProc := object.DefaultProcessor(object.ToService, nil)(svcIdx, cache.ResourceEventHandlerFuncs{})
+	epProc := object.DefaultProcessor(object.EndpointSliceToEndpoints, dns.EndpointSliceLatencyRecorder())(epIdx, cache.ResourceEventHandlerFuncs{})
+
+	object.DurationSinceFunc = func(t time.Time) time.Duration {
 		return now.Sub(t)
 	}
-	DNSProgrammingLatency.Reset()
-	go controller.Run()
+	object.DNSProgrammingLatency.Reset()
 
 	endpoints1 := []discovery.Endpoint{{
 		Addresses: []string{"1.2.3.4"},
@@ -74,44 +68,40 @@ func TestDNSProgrammingLatencyEndpointSlices(t *testing.T) {
 		Addresses: []string{"1.2.3.45"},
 	}}
 
-	createService(t, client, controller, "my-service", api.ClusterIPNone)
-	createEndpointSlice(t, client, "my-service", now.Add(-2*time.Second), endpoints1)
-	updateEndpointSlice(t, client, "my-service", now.Add(-1*time.Second), endpoints2)
-
-	createEndpointSlice(t, client, "endpoints-no-service", now.Add(-4*time.Second), nil)
+	createService(t, svcProc, "my-service", api.ClusterIPNone)
+	createEndpointSlice(t, epProc, "my-service", now.Add(-2*time.Second), endpoints1)
+	updateEndpointSlice(t, epProc, "my-service", now.Add(-1*time.Second), endpoints2)
 
-	createService(t, client, controller, "clusterIP-service", "10.40.0.12")
-	createEndpointSlice(t, client, "clusterIP-service", now.Add(-8*time.Second), nil)
+	createEndpointSlice(t, epProc, "endpoints-no-service", now.Add(-4*time.Second), nil)
 
-	createService(t, client, controller, "headless-no-annotation", api.ClusterIPNone)
-	createEndpointSlice(t, client, "headless-no-annotation", nil, nil)
+	createService(t, svcProc, "clusterIP-service", "10.40.0.12")
+	createEndpointSlice(t, epProc, "clusterIP-service", now.Add(-8*time.Second), nil)
 
-	createService(t, client, controller, "headless-wrong-annotation", api.ClusterIPNone)
-	createEndpointSlice(t, client, "headless-wrong-annotation", "wrong-value", nil)
+	createService(t, svcProc, "headless-no-annotation", api.ClusterIPNone)
+	createEndpointSlice(t, epProc, "headless-no-annotation", nil, nil)
 
-	controller.Stop()
+	createService(t, svcProc, "headless-wrong-annotation", api.ClusterIPNone)
+	createEndpointSlice(t, epProc, "headless-wrong-annotation", "wrong-value", nil)
 
-	if err := testutil.CollectAndCompare(DNSProgrammingLatency, strings.NewReader(expected)); err != nil {
+	if err := testutil.CollectAndCompare(object.DNSProgrammingLatency, strings.NewReader(expected)); err != nil {
 		t.Error(err)
 	}
 }
 
 func TestDnsProgrammingLatencyEndpoints(t *testing.T) {
-	client := fake.NewSimpleClientset()
 	now := time.Now()
-	ctx := context.TODO()
-	controller := newdnsController(ctx, client, dnsControlOpts{
-		initEndpointsCache: true,
-		useEndpointSlices:  false,
-		// This is needed as otherwise the fake k8s client doesn't work properly.
-		skipAPIObjectsCleanup: true,
-	})
-
-	durationSinceFunc = func(t time.Time) time.Duration {
+
+	svcIdx := cache.NewIndexer(cache.DeletionHandlingMetaNamespaceKeyFunc, cache.Indexers{svcNameNamespaceIndex: svcNameNamespaceIndexFunc})
+	epIdx := cache.NewIndexer(cache.DeletionHandlingMetaNamespaceKeyFunc, cache.Indexers{})
+
+	dns := dnsControl{svcLister: svcIdx}
+	svcProc := object.DefaultProcessor(object.ToService, nil)(svcIdx, cache.ResourceEventHandlerFuncs{})
+	epProc := object.DefaultProcessor(object.ToEndpoints, dns.EndpointsLatencyRecorder())(epIdx, cache.ResourceEventHandlerFuncs{})
+
+	object.DurationSinceFunc = func(t time.Time) time.Duration {
 		return now.Sub(t)
 	}
-	DNSProgrammingLatency.Reset()
-	go controller.Run()
+	object.DNSProgrammingLatency.Reset()
 
 	subset1 := []api.EndpointSubset{{
 		Addresses: []api.EndpointAddress{{IP: "1.2.3.4", Hostname: "foo"}},
@@ -121,24 +111,22 @@ func TestDnsProgrammingLatencyEndpoints(t *testing.T) {
 		Addresses: []api.EndpointAddress{{IP: "1.2.3.5", Hostname: "foo"}},
 	}}
 
-	createService(t, client, controller, "my-service", api.ClusterIPNone)
-	createEndpoints(t, client, "my-service", now.Add(-2*time.Second), subset1)
-	updateEndpoints(t, client, "my-service", now.Add(-1*time.Second), subset2)
-
-	createEndpoints(t, client, "endpoints-no-service", now.Add(-4*time.Second), nil)
+	createService(t, svcProc, "my-service", api.ClusterIPNone)
+	createEndpoints(t, epProc, "my-service", now.Add(-2*time.Second), subset1)
+	updateEndpoints(t, epProc, "my-service", now.Add(-1*time.Second), subset2)
 
-	createService(t, client, controller, "clusterIP-service", "10.40.0.12")
-	createEndpoints(t, client, "clusterIP-service", now.Add(-8*time.Second), nil)
+	createEndpoints(t, epProc, "endpoints-no-service", now.Add(-4*time.Second), nil)
 
-	createService(t, client, controller, "headless-no-annotation", api.ClusterIPNone)
-	createEndpoints(t, client, "headless-no-annotation", nil, nil)
+	createService(t, svcProc, "clusterIP-service", "10.40.0.12")
+	createEndpoints(t, epProc, "clusterIP-service", now.Add(-8*time.Second), nil)
 
-	createService(t, client, controller, "headless-wrong-annotation", api.ClusterIPNone)
-	createEndpoints(t, client, "headless-wrong-annotation", "wrong-value", nil)
+	createService(t, svcProc, "headless-no-annotation", api.ClusterIPNone)
+	createEndpoints(t, epProc, "headless-no-annotation", nil, nil)
 
-	controller.Stop()
+	createService(t, svcProc, "headless-wrong-annotation", api.ClusterIPNone)
+	createEndpoints(t, epProc, "headless-wrong-annotation", "wrong-value", nil)
 
-	if err := testutil.CollectAndCompare(DNSProgrammingLatency, strings.NewReader(expected)); err != nil {
+	if err := testutil.CollectAndCompare(object.DNSProgrammingLatency, strings.NewReader(expected)); err != nil {
 		t.Error(err)
 	}
 }
@@ -175,49 +163,41 @@ func buildEndpointSlice(name string, lastChangeTriggerTime interface{}, endpoint
 	}
 }
 
-func createEndpoints(t *testing.T, client kubernetes.Interface, name string, triggerTime interface{}, subsets []api.EndpointSubset) {
-	ctx := context.TODO()
-	_, err := client.CoreV1().Endpoints(namespace).Create(ctx, buildEndpoints(name, triggerTime, subsets), meta.CreateOptions{})
+func createEndpoints(t *testing.T, processor cache.ProcessFunc, name string, triggerTime interface{}, subsets []api.EndpointSubset) {
+	err := processor(cache.Deltas{{Type: cache.Added, Object: buildEndpoints(name, triggerTime, subsets)}})
 	if err != nil {
 		t.Fatal(err)
 	}
 }
 
-func updateEndpoints(t *testing.T, client kubernetes.Interface, name string, triggerTime interface{}, subsets []api.EndpointSubset) {
-	ctx := context.TODO()
-	_, err := client.CoreV1().Endpoints(namespace).Update(ctx, buildEndpoints(name, triggerTime, subsets), meta.UpdateOptions{})
+func updateEndpoints(t *testing.T, processor cache.ProcessFunc, name string, triggerTime interface{}, subsets []api.EndpointSubset) {
+	err := processor(cache.Deltas{{Type: cache.Updated, Object: buildEndpoints(name, triggerTime, subsets)}})
 	if err != nil {
 		t.Fatal(err)
 	}
 }
 
-func createEndpointSlice(t *testing.T, client kubernetes.Interface, name string, triggerTime interface{}, endpoints []discovery.Endpoint) {
-	ctx := context.TODO()
-	_, err := client.DiscoveryV1beta1().EndpointSlices(namespace).Create(ctx, buildEndpointSlice(name, triggerTime, endpoints), meta.CreateOptions{})
+func createEndpointSlice(t *testing.T, processor cache.ProcessFunc, name string, triggerTime interface{}, endpoints []discovery.Endpoint) {
+	err := processor(cache.Deltas{{Type: cache.Added, Object: buildEndpointSlice(name, triggerTime, endpoints)}})
 	if err != nil {
 		t.Fatal(err)
 	}
 }
 
-func updateEndpointSlice(t *testing.T, client kubernetes.Interface, name string, triggerTime interface{}, endpoints []discovery.Endpoint) {
-	ctx := context.TODO()
-	_, err := client.DiscoveryV1beta1().EndpointSlices(namespace).Update(ctx, buildEndpointSlice(name, triggerTime, endpoints), meta.UpdateOptions{})
+func updateEndpointSlice(t *testing.T, processor cache.ProcessFunc, name string, triggerTime interface{}, endpoints []discovery.Endpoint) {
+	err := processor(cache.Deltas{{Type: cache.Updated, Object: buildEndpointSlice(name, triggerTime, endpoints)}})
 	if err != nil {
 		t.Fatal(err)
 	}
 }
 
-func createService(t *testing.T, client kubernetes.Interface, controller dnsController, name string, clusterIp string) {
-	ctx := context.TODO()
-	if _, err := client.CoreV1().Services(namespace).Create(ctx, &api.Service{
+func createService(t *testing.T, processor cache.ProcessFunc, name string, clusterIp string) {
+	obj := &api.Service{
 		ObjectMeta: meta.ObjectMeta{Namespace: namespace, Name: name},
 		Spec:       api.ServiceSpec{ClusterIP: clusterIp},
-	}, meta.CreateOptions{}); err != nil {
-		t.Fatal(err)
 	}
-	if err := wait.PollImmediate(10*time.Millisecond, 10*time.Second, func() (bool, error) {
-		return len(controller.SvcIndex(object.ServiceKey(name, namespace))) == 1, nil
-	}); err != nil {
+	err := processor(cache.Deltas{{Type: cache.Added, Object: obj}})
+	if err != nil {
 		t.Fatal(err)
 	}
 }
diff --git a/test/file_upstream_test.go b/test/file_upstream_test.go
index 4d13a2ca0..6b796f90d 100644
--- a/test/file_upstream_test.go
+++ b/test/file_upstream_test.go
@@ -58,6 +58,113 @@ www 3600 IN CNAME www.example.net.
 	}
 }
 
+func TestFileUpstreamError(t *testing.T) {
+	cases := map[string]test.Case{
+		"nxdomain": {
+			Qname: "nxdomain.example.org.", Qtype: dns.TypeA,
+			Answer: []dns.RR{
+				test.CNAME("nxdomain.example.org.	3600	IN	CNAME	nxdomain.example.net"),
+			},
+			Rcode: dns.RcodeNameError,
+		},
+		"nxdomain-chain": {
+			Qname: "chain1.example.org.", Qtype: dns.TypeA,
+			Answer: []dns.RR{
+				test.CNAME("chain1.example.org.	3600	IN	CNAME	nxdomain.example.org"),
+				test.CNAME("nxdomain.example.org.	3600	IN	CNAME	nxdomain.example.net"),
+			},
+			Rcode: dns.RcodeNameError,
+		},
+		"srvfail": {
+			Qname: "srvfail.example.org.", Qtype: dns.TypeA,
+			Rcode: dns.RcodeServerFailure,
+		},
+		"srvfail-chain": {
+			Qname: "chain2.example.org.", Qtype: dns.TypeA,
+			Rcode: dns.RcodeServerFailure,
+		},
+		"nodata": {
+			Qname: "nodata.example.org.", Qtype: dns.TypeA,
+			Answer: []dns.RR{
+				test.CNAME("nodata.example.org.	3600	IN	CNAME	nodata.example.net"),
+			},
+			Rcode: dns.RcodeSuccess,
+		},
+		"nodata-chain": {
+			Qname: "chain3.example.org.", Qtype: dns.TypeA,
+			Answer: []dns.RR{
+				test.CNAME("chain3.example.org.	3600	IN	CNAME	nodata.example.org"),
+				test.CNAME("nodata.example.org.	3600	IN	CNAME	nodata.example.net"),
+			},
+			Rcode: dns.RcodeSuccess,
+		},
+	}
+	name, rm, err := test.TempFile(".", `$ORIGIN example.org.
+@	3600 IN	SOA   sns.dns.icann.org. noc.dns.icann.org. (
+        2017042745 ; serial
+        7200       ; refresh (2 hours)
+        3600       ; retry (1 hour)
+        1209600    ; expire (2 weeks)
+        3600       ; minimum (1 hour)
+)
+
+    3600 IN NS    a.iana-servers.net.
+    3600 IN NS    b.iana-servers.net.
+
+chain1   3600 IN CNAME nxdomain
+nxdomain 3600 IN CNAME nxdomain.example.net.
+chain2   3600 IN CNAME srvfail
+srvfail  3600 IN CNAME srvfail.example.net.
+chain3   3600 IN CNAME nodata
+nodata   3600 IN CNAME nodata.example.net.
+
+`)
+	if err != nil {
+		t.Fatalf("Failed to create zone: %s", err)
+	}
+	defer rm()
+
+	corefile := `.:0 {
+	template ANY A nxdomain.example.net. {
+		rcode NXDOMAIN
+	}
+	template ANY A srvfail.example.net. {
+		rcode SERVFAIL
+	}
+	template ANY A nodata.example.net. {
+	}
+	file ` + name + ` example.org
+}`
+
+	i, udp, _, err := CoreDNSServerAndPorts(corefile)
+	if err != nil {
+		t.Fatalf("Could not get CoreDNS serving instance: %s", err)
+	}
+	defer i.Stop()
+
+	for n, tc := range cases {
+		t.Run(n, func(t *testing.T) {
+			m := new(dns.Msg)
+			m.SetQuestion(tc.Qname, tc.Qtype)
+			m.SetEdns0(4096, true)
+
+			r, err := dns.Exchange(m, udp)
+			if err != nil {
+				t.Fatalf("Could not exchange msg: %s", err)
+			}
+			if r.Rcode != tc.Rcode {
+				t.Fatalf("expected rcode %v, got %v", tc.Rcode, r.Rcode)
+			}
+			if n := len(r.Answer); n != len(tc.Answer) {
+				t.Fatalf("Expected %v answers, got %v", len(tc.Answer), n)
+			}
+			if err := test.Section(tc, test.Answer, r.Answer); err != nil {
+				t.Error(err)
+			}
+		})
+	}
+}
+
 // TestFileUpstreamAdditional runs two CoreDNS servers that serve example.org and foo.example.org.
 // example.org contains a cname to foo.example.org; this should be resolved via upstream.Self.
 func TestFileUpstreamAdditional(t *testing.T) {
diff --git a/test/go-test-tmpfile565156097 b/test/go-test-tmpfile565156097
new file mode 100644
index 000000000..752ec30cf
--- /dev/null
+++ b/test/go-test-tmpfile565156097
@@ -0,0 +1,19 @@
+$ORIGIN example.org.
+@	3600 IN	SOA   sns.dns.icann.org. noc.dns.icann.org. (
+        2017042745 ; serial
+        7200       ; refresh (2 hours)
+        3600       ; retry (1 hour)
+        1209600    ; expire (2 weeks)
+        3600       ; minimum (1 hour)
+)
+
+    3600 IN NS    a.iana-servers.net.
+    3600 IN NS    b.iana-servers.net.
+
+chain1   3600 IN CNAME nxdomain
+nxdomain 3600 IN CNAME nxdomain.example.net.
+chain2   3600 IN CNAME srvfail
+srvfail  3600 IN CNAME srvfail.example.net.
+chain3   3600 IN CNAME nodata
+nodata   3600 IN CNAME nodata.example.net.
+