jiebi/CodeConvo huggingface dataset:
Code change:
fi # Create coverage report directories. <del> KUBE_TEST_API_HASH="$(echo -n "${KUBE_TEST_API////-}"| ${SHA1SUM} |awk '{print $1}')" </del> if [[ -z "${KUBE_COVER_REPORT_DIR}" ]]; then <del> cover_report_dir="/tmp/k8s_coverage/${KUBE_TEST_API_HASH}/$(kube::util::sortable_date)" </del> <ins> cover_report_dir="/tmp/k8s_coverage/$(kube::util::sortable_date)" </ins> else cover_report_dir="${KUBE_COVER_REPORT_DIR}" fi

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: unit tests with coverage succeed How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Looking at : It is expecting group version string.
fixed in , clean up from
Code change:
wantToRun, shouldSchedule, shouldContinueRunning, err := manager.nodeShouldRunDaemonPod(node, c.ds) if wantToRun != c.wantToRun { <del> t.Errorf("[%v] expected wantToRun: %v, got: %v", i, c.wantToRun, wantToRun) </del> <ins> t.Errorf("[%v] strategy: %v, predicateName: %v expected wantToRun: %v, got: %v", i, c.ds.Spec.UpdateStrategy.Type, c.predicateName, c.wantToRun, wantToRun) </ins> } if shouldSchedule != c.shouldSchedule { <del> t.Errorf("[%v] expected shouldSchedule: %v, got: %v", i, c.shouldSchedule, shouldSchedule) </del> <ins> t.Errorf("[%v] strategy: %v, predicateName: %v expected shouldSchedule: %v, got: %v", i, c.ds.Spec.UpdateStrategy.Type, c.predicateName, c.shouldSchedule, shouldSchedule) </ins> } if shouldContinueRunning != c.shouldContinueRunning { <del> t.Errorf("[%v] expected shouldContinueRunning: %v, got: %v", i, c.shouldContinueRunning, shouldContinueRunning) </del> <ins> t.Errorf("[%v] strategy: %v, predicateName: %v expected shouldContinueRunning: %v, got: %v", i, c.ds.Spec.UpdateStrategy.Type, c.predicateName, c.shouldContinueRunning, shouldContinueRunning) </ins> } if err != c.err { <del> t.Errorf("[%v] expected err: %v, got: %v", i, c.err, err) </del> <ins> t.Errorf("[%v] strategy: %v, predicateName: %v expected err: %v, got: %v", i, c.predicateName, c.ds.Spec.UpdateStrategy.Type, c.err, err) </ins> } } }

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: /kind enhancement /sig apps Description: is a key function of DaemonSet; it's better to have more unit test case for it, e.g. node status, pod on the node. There's several test cases to the DaemonSet's finial decision, which are kind of integration test to me; prefer to provide more unit test case for this key function.
GitHub didn't allow me to assign the following users: mathspanda. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
do you have any specific cases or shall I figure out something? I can take a look at this issue if you don't mind.
, , thanks very much for our effort on this enhancement; as you known, DaemonSet will use scheduler's predicates to decide which node can place pods; there's several predicates errors , the purpose of this issue is to trigger every error in unit test for code coverage.
Sure, thanks. I'll cover these predicates.
Also I've found interesting issue when I wanted to add test for predicate. It looks like is not handled or maybe I did something wrong, I couldn't find check. According following comments it should fail because AWS EBS forbids any two pods mounting the same volume ID : Test case: Any thoughts? is it correct behavior?
Code change:
// ListenAndServePodResources runs the kubelet podresources grpc service func (kl *Kubelet) ListenAndServePodResources() { <del> server.ListenAndServePodResources(util.LocalEndpoint(kl.getPodResourcesDir(), podresources.Socket), kl.podManager, kl.containerManager) </del> <ins> socket, err := util.LocalEndpoint(kl.getPodResourcesDir(), podresources.Socket) if err != nil { klog.V(2).Infof("Failed to get local endpoint for PodResources endpoint: %v", err) return } server.ListenAndServePodResources(socket, kl.podManager, kl.containerManager) </ins> } // Delete the eligible dead container instances in a pod. Depending on the configuration, the latest dead containers may be kept around.

Related discussion:
text: What happened: I replaced the kubelet on one of my 1.14 nodes with a build from master, and now I'm getting this error starting up. Test passes since 5/31/2019 commit have been failing on testgrid. If you look at the logs, the Windows nodes are not coming up. What you expected to happen: No errors starting up kubelet How to reproduce it (as minimally and precisely as possible): Build master, then try to run the kubelet on Windows Anything else we need to know?: /sig windows Environment: Kubernetes version (use ): Cloud provider or hardware configuration: Azure OS (e.g: ): Windows Server 2019 Kernel (e.g. ): 10.0..529 Install tools: aks-engine
cc /priority critical-urgent
/milestone v1.15 Release team - it looks like a recent PR broke windows nodes. We will need a fix here for 1.15
Ok, so the problem here is a little more complicated and we're actually seeing a bug that is actually in the code for quite some time, just that we never used that particular code path. But first: TL;DR: Quick fix is to start the kubelet on Windows with: --feature-gates="KubeletPodResources=false" Now, as to why this is happening: PR [1] was merged 3 days ago and it graduates KubeletPodResources feature as Beta. That means kubelet will enable it by default.With the introduction of the above feature, this branch of code is entered when starting kubelet [2]. We can see a call to the function: defined here [3].While innocent enough, gets the necessary npipe by calling: . is defined here [4] for Windows . The problem is actually that url.URL can't handle Windows style paths [5] and will replace "" with "%5C" as it's to be expected. The paths are transformed along the way from the default "/var/lib/kubelet" to "varlibkubelet" by defined here [6]. More specifically in the use of to create a path. It's inputs are unix style but returns the path Windows style as it's to be expected.The solution is to replace the usage of url.URL in with some alternative package or just implement it.
Code change:
// ListenAndServePodResources runs the kubelet podresources grpc service func (kl *Kubelet) ListenAndServePodResources() { <del> server.ListenAndServePodResources(util.LocalEndpoint(kl.getPodResourcesDir(), podresources.Socket), kl.podManager, kl.containerManager) </del> <ins> socket, err := util.LocalEndpoint(kl.getPodResourcesDir(), podresources.Socket) if err != nil { klog.V(2).Infof("Failed to get local endpoint for PodResources endpoint: %v", err) return } server.ListenAndServePodResources(socket, kl.podManager, kl.containerManager) </ins> } // Delete the eligible dead container instances in a pod. Depending on the configuration, the latest dead containers may be kept around.

Related discussion:
text: [1] [2] [3] [4] [5] [6]
Thanks for the analysis! I was a bit concerned around the fact that this started showing up around my commit: but the root cause here is indeed, as you said, the graduation of feature to Beta and the way the invalid escape sequences are being introduced.
GCE tests failing too /cc
I'm trying a quick to see if anything else is broken. There are other functions in that same file doing that substitution already so it may make sense to use that as a workaround until a better solution is available.
Still working on it - looks like the path to create the directories is also busted. With still getting
Code change:
github.com/lpabon/godbc v0.1.1 // indirect github.com/magiconair/properties v1.8.1 // indirect github.com/miekg/dns v1.1.4 <del> github.com/moby/ipvs v1.0.0 </del> <ins> github.com/moby/ipvs v1.0.1 </ins> github.com/mohae/deepcopy v0.0.0-20170603005431-491d3605edfb // indirect github.com/mrunalp/fileutils v0.0.0-20171103030105-7d4729fb3618 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822

Related discussion:
text: Hi, Since my upgrade to 1.18 version, I have errors in kube-proxy: I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors. Do you have also this issue ? OS: RHEL7 Kubernetes version (use ):
/sig network
Related to cc
/cc
Taking one of the outputs it gives me the following addresses? can you share more details about your environment, Is it possible that the CoreDNS pod has the address ? what are your cluster-cidr and service-cidr? Indeed that new net.Utils change looks suspicious , a quick grep in the repo gives me some suspects, maybe the service/ipallocator? It's important to mention there were similar issues with IPVS before, interpreting IPv4 addresses as IPv6, has some experience with this kind of issues, maybe he can see something:) /cc
A bit tricky to trace since libnetwork/ipvs is moving around at the moment. Problem seems to be in; The "family" must be wrong. It is called from;
Please do a and attach the output.
Hi, Some informations: cluster-cidr and service-cidr: fd00:10:245:0:0:0:0:0/64,10.245.0.0/16 fd00:10:96:0:0:0:0:0/112,10.96.0.0/12 ipvsadm output:
Ref:
The commit in the ref above should be in v1.18.0 which is the version used in this issue. So the commit does not fix the problem.
Maybe something wrong around here Before , it looks like this: Now it looks like this: However is not initializedit is zero :(.
I do not get these faults. Has the netlink API changed? What kernel version are you using?
d.AddressFamily is supposed to be set from the "NetlinkRouteAttr" slice; It should definitely not be hard-coded to ipv4.
Made a trying to reproduce this in my environment but didn't got this error.
Code change:
github.com/lpabon/godbc v0.1.1 // indirect github.com/magiconair/properties v1.8.1 // indirect github.com/miekg/dns v1.1.4 <del> github.com/moby/ipvs v1.0.0 </del> <ins> github.com/moby/ipvs v1.0.1 </ins> github.com/mohae/deepcopy v0.0.0-20170603005431-491d3605edfb // indirect github.com/mrunalp/fileutils v0.0.0-20171103030105-7d4729fb3618 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822

Related discussion:
text: I may try this on some older kernel like from CentOS 7 Added exactly the same IPVS configuration here: And when running this 'gist/program' pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result: Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.
After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on . Accroding to the code, netlink will try to get attribute, but I find the kernel just does not support it :( This is the defined in (kernel 3.10) No attribute is defined! But in new kernel version, the is defined like : Obviously, the kernel has some attributes (IPVSDESTATTRADDRFAMILY, IPVSDESTATTRSTATS64...). That is why kube-proxy works well on systems with a higher version of the kernel. So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.
Here is the defined in :
Good finding. So, we can conclude that this really is kernel version dependent. It is hard to know what to do about it. We must check the family. I would propose that we make it work for ipv4 with some fallback and put a requirement on kernel version for ipv6-only/dual-stack. But the decision is others to make. /cc
/cc
The symbol in the kernel tree The for service family seems very old. Since the service must have the same family as the real-servers perhaps it can be used for compatibility with old kernels?
The commit that adds the symbol; Seem to go into linux-3.18
I did not know the minimum required for the kernel version, I was using the stable and official version of RHEL7. My kernel version is: Currently the max version in the repository is: According to the documentation (), I should switch to RHEL8 if I want a recent kernel :-
There is no minimum kernel version requirement afaik. So I think this must be fixed for ipv4-only clusters IMO.
Code change:
github.com/lpabon/godbc v0.1.1 // indirect github.com/magiconair/properties v1.8.1 // indirect github.com/miekg/dns v1.1.4 <del> github.com/moby/ipvs v1.0.0 </del> <ins> github.com/moby/ipvs v1.0.1 </ins> github.com/mohae/deepcopy v0.0.0-20170603005431-491d3605edfb // indirect github.com/mrunalp/fileutils v0.0.0-20171103030105-7d4729fb3618 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822

Related discussion:
text: However ipv6-only and dual-stack are in "alpha" so a requirement on kernel version might be ok, but I can't say myself. A complication is that the problem is not in k8s itself but in a 3rd party lib.
ipv6-only is beta since 1.18 :smile:
/assign
I opened a PR to fix this in moby/ipvs If someone w/ a cluster on the older kernel version can test it that would be great, otherwise I will setup a cluster to validate this some time this week.
Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I've mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now
with a small test, it worked fine. Environment: CentOS 7.1 - Kernel 3.10.0-229.el7.x86_64 + ipvsadm 1.2.1 Pre-loaded some IPv6 rules into ipvs: I've used code, searching for the following structure: I then call ` to get the IPVS object, and then try to fetch the destinations. The following are the results: Before patching the ipvs module: After patching the ipvs module (pinning to your repo/branch) Also I've made those tests with the structure but searching for TCP, and for kube-apiserver (that have only one destination) and it worked fine in all those scenarios. If wanted I can improve the gist to pass the parameters as command arguments instead of hardcoded (sorry, did this in a rush to test)
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
/remove-triage unresolved
Thanks just waiting on now
same issue,sloved by update kernel to 5.X
I didn't want to wait, I updated to kernel-lt in the version 4.4 . The problem is solved on my side :)
Thanks for confirming. We'll still try to get in since I'm sure other folks with older kernels will run into this issue.
/retitle parseIP error with IPVS proxy
upgrade your system kernel to 4.xx edition
FYI: just merged, need to run some validation & cut a new release (v1.0.1).
Code change:
github.com/lpabon/godbc v0.1.1 // indirect github.com/magiconair/properties v1.8.1 // indirect github.com/miekg/dns v1.1.4 <del> github.com/moby/ipvs v1.0.0 </del> <ins> github.com/moby/ipvs v1.0.1 </ins> github.com/mohae/deepcopy v0.0.0-20170603005431-491d3605edfb // indirect github.com/mrunalp/fileutils v0.0.0-20171103030105-7d4729fb3618 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822

Related discussion:
text: FYI
/retitle DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7
v1.18 cherry-pick
Hi, it seems that you're using v1.18.0 and this has been corrected in v1.18.3, can you please update and check? Tks
Code change:
}, func(client kubeclientset.Interface, obj pkgruntime.Object) error { svc := obj.(*v1.Service) <del> err := client.Core().Services(svc.Namespace).Delete(svc.Name, &metav1.DeleteOptions{}) </del> <ins> orphanDependents := false err := client.Core().Services(svc.Namespace).Delete(svc.Name, &metav1.DeleteOptions{OrphanDependents: &orphanDependents}) </ins> return err })

Related discussion:
text: Problem: When users sends a cascading deletion request to federation i.e they set orphanDependents=false, then federation should also set orphanDependents=false, when it sends the request to underlying clusters.
Fixed by
Code change:
"github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/tools" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util" </ins> "github.com/coreos/go-etcd/etcd" "github.com/golang/glog" )

Related discussion:
text: kube-proxy regularly stops running with the below panic / stack-trace. This is when configured with vagrant. (It is also running on a laptop, so my unsupported-by-any-evidence theory is that it happens when the laptop resumes after sleep.) Restarting the service causes the proxy to continue working happily. (Should systemd be restarting the service?) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: panic: runtime error: invalid memory address or nil pointer dereference Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: [signal 0xb code=0x1 addr=0x0 pc=0x431b51] Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 22 [running]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0x6b7c00, 0xa51de8) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /usr/local/go/src/pkg/runtime/panic.c:266 +0xb6 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210066600, 0xc210037660, 0xc210037720, 0x77359400, 0x0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x841 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210066600, 0xc210037660, 0xc210037720,
Code change:
"github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/tools" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util" </ins> "github.com/coreos/go-etcd/etcd" "github.com/golang/glog" )

Related discussion:
text: 0x77359400) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x165 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x1b3 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 1 [select (no cases)]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: () Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x47a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 3 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (loggingT).flushDaemon(0xa57340) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x50 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by ·1 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x276 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 4 [syscall]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: () Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /usr/local/go/src/pkg/runtime/proc.c:1394 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 6 [sleep]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: time.Sleep(0x12a05f200) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: +0x31 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0x7aca18,
Code change:
"github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/tools" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util" </ins> "github.com/coreos/go-etcd/etcd" "github.com/golang/glog" )

Related discussion:
text: 0x12a05f200) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x57 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x99 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 7 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210077120, 0x7ff2602627f0, 0xc210077060, 0xc210037540) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x196 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 8 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210077210, 0x7ff260262840, 0xc210077150, 0xc2100375a0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x196 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 9 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (Mux).listen(0xc2100770c0, 0x718260, 0x4,
Code change:
"github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/tools" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util" </ins> "github.com/coreos/go-etcd/etcd" "github.com/golang/glog" )

Related discussion:
text: 0xc210037600) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·001() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x43 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·001() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x40 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc21000a920, 0x0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x49 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by (*Mux).Channel Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x20a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 10 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·004() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x4d Aug 13 12:04:06 kubernetes-minion-1 systemd[1]: kube-proxy.service: main process exited, code=exited, status=2/INVALIDARGUMENT Aug 13 12:04:06 kubernetes-minion-1 systemd[1]: Unit kube-proxy.service entered failed state.
I looked at this one a little bit and I think it's because of This line isn't present at , but I'm not sure which is newer. Presumably results in the channel being closed (and receiving a zero value that it then tries to dereference) on error.
Diagnostic: This line doesn't check for a closed channel, and it needs to. Yeah, how are we supposed to know if our client is going to close the channel or not?
Code change:
"github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/tools" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util" </ins> "github.com/coreos/go-etcd/etcd" "github.com/golang/glog" )

Related discussion:
text: I think the behavior has changed at least once already. This is part of why I'm trying to (for our code, anyway) standardize all of our watches with a watch.Interface. I took the liberty of assigning this to you since you're more familiar with this code, but feel free to unassign if you can't take it right now.
Got it
can you give a try?
after i do the minion watch for kubelet config i'll come back and gut this and replace it.
Thanks - with merged I haven't seen this happening. Thanks for the super-quick fix!