jiebi/CodeConvo huggingface dataset:
Code change:
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite} Happened on a presubmit run in .
Sorry, it is a new e2e test I am still working on. It is not merged yet.
Don't worry about it, our flake detection isn't quite perfect yet :)
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention.
This new e2e test has been merged. Let's observe it few more days and close this issue if nothing bad happen.
[FLAKE-PING] This flaky-test issue would love to have more attention.
This does't look like a failure in new test, rather its a generic failure with serviceaccounts.
Thanks for the investigation. Yeah I think it is the serviceaccounts issue. I wonder when could we close this flaky issue. Seems like it is stable if serviceaccounts issue is fixed.
You can close it with a reference to the serviceaccounts issue - do you know the # for that one ?
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
Have a fix for this, I'm now keeping running tests on my own cluster. Will send out PR once verify the fix works.

Related discussion:
text: Expect(err).NotTo(HaveOccurred()) }() <del> // Waiting for service to expose endpoint </del> <ins> // Waiting for service to expose endpoint. </ins> validateEndpointsOrFail(c, ns, serviceName, PortsByPodName{serverPodName: {servicePort}}) By("Retrieve sourceip from a pod on the same node")
Code change:
On a cluster using GCI, I'm not able to create docker containers using the docker CLI. On further introspection, it looks like bridge is being deleted. I consider this a regression. K8s used to allow creating docker containers out of band and in-fact it is a feature that is taken into account as part of k8s node features. To further exacerbate this issue, exists on containervm. We need to match the behavior between GCI and container-vm and ideally stop deleting docker0 bridge. cc
During GCI start up, got deleted to avoid overlapping cidr with . ref:
there's no need to delete docker0 afaik. It was deleted on gci master to fix the "overlapping cidr" problem, i.e we were supplying to kubelet on GKE master, and this resulted in 2 bridges with the same cidr. At the time I'd suggested passing the correct cidr. Don't remember where this discussion ended. Note that containers on docker0 with/without cni are broken anyway, we used to start docker with --bridge=cbr0, so would result in an ip from cbr0. This is no longer the case. We use kubenet to put containers on cbr0, so will result in an ip from docker0, which is not connected to nanything else.
This shouldn't cause docker crashes? you should still be able to create containers, just no networking. And networking is broken through docker CLI anyway because of the cni issue mentioned, or are you observing something different?
will return error. Have to add
Since bridge has been deleted, docker run by default will fail unless one overrides the network ns to be host or none.
so what we're really losing is the ability to docker run then talk to others (external internet, other pods) without net=host on GCI. This is useful and we should fix it, as mentioned earlier we probably don't need to delete docker0. docker run + others talking to us is broken because of the cni change, and will be broken even with docker0 (IMO this is fine).
cc
Yeah. I'd prefer ignoring docker0 as long as it doesn't affect kubelet functionality. , Prashanth B wrote:
Vish pointed out that of docker dies, docker0 comes back. Given that we should be consistent. Either force docker not to create a bridge at all, EVER, or just ignore docker0. We may even want to consider assigning --bip= 169.254.123.0/24 or something, so we can reliably use 172.* for clusters eventually. , Vish Kannan wrote:

Related discussion:
text: docker_opts+=" --log-level=warn" fi local use_net_plugin="true" <del> if [[ "${NETWORK_PROVIDER:-}" != "kubenet" && "${NETWORK_PROVIDER:-}" != "cni" ]]; then </del> <ins> if [[ "${NETWORK_PROVIDER:-}" == "kubenet" || "${NETWORK_PROVIDER:-}" == "cni" ]]; then # set docker0 cidr to private ip address range to avoid conflict with cbr0 cidr range docker_opts+=" --bip=169.254.123.1/24" else </ins> use_net_plugin="false" docker_opts+=" --bridge=cbr0" fi
Code change:
On a cluster using GCI, I'm not able to create docker containers using the docker CLI. On further introspection, it looks like bridge is being deleted. I consider this a regression. K8s used to allow creating docker containers out of band and in-fact it is a feature that is taken into account as part of k8s node features. To further exacerbate this issue, exists on containervm. We need to match the behavior between GCI and container-vm and ideally stop deleting docker0 bridge. cc
During GCI start up, got deleted to avoid overlapping cidr with . ref:
there's no need to delete docker0 afaik. It was deleted on gci master to fix the "overlapping cidr" problem, i.e we were supplying to kubelet on GKE master, and this resulted in 2 bridges with the same cidr. At the time I'd suggested passing the correct cidr. Don't remember where this discussion ended. Note that containers on docker0 with/without cni are broken anyway, we used to start docker with --bridge=cbr0, so would result in an ip from cbr0. This is no longer the case. We use kubenet to put containers on cbr0, so will result in an ip from docker0, which is not connected to nanything else.
This shouldn't cause docker crashes? you should still be able to create containers, just no networking. And networking is broken through docker CLI anyway because of the cni issue mentioned, or are you observing something different?
will return error. Have to add
Since bridge has been deleted, docker run by default will fail unless one overrides the network ns to be host or none.
so what we're really losing is the ability to docker run then talk to others (external internet, other pods) without net=host on GCI. This is useful and we should fix it, as mentioned earlier we probably don't need to delete docker0. docker run + others talking to us is broken because of the cni change, and will be broken even with docker0 (IMO this is fine).
cc
Yeah. I'd prefer ignoring docker0 as long as it doesn't affect kubelet functionality. , Prashanth B wrote:
Vish pointed out that of docker dies, docker0 comes back. Given that we should be consistent. Either force docker not to create a bridge at all, EVER, or just ignore docker0. We may even want to consider assigning --bip= 169.254.123.0/24 or something, so we can reliably use 172.* for clusters eventually. , Vish Kannan wrote:

Related discussion:
text: WantedBy=multi-user.target EOF <del> # Delete docker0 to avoid interference </del> <ins> # Flush iptables nat table </ins> iptables -t nat -F || true <del> ip link set docker0 down || true brctl delbr docker0 || true </del> systemctl start kubelet.service }
Code change:
When I tried to use the following command to build kubernetes: I encountered the following errors: I'm using Ubuntu 14.04 and the code from master branch. will submit a patch then to fix this problem.
whoa, I just landed in the exact same place, I am trying to build k8s on a CentOS7 box. Tried with removing the unused utils and works like a charm !

Related discussion:
text: "os" "k8s.io/kubernetes/pkg/healthz" <del> "k8s.io/kubernetes/pkg/util" </del> "k8s.io/kubernetes/pkg/util/flag" "k8s.io/kubernetes/pkg/util/logs" "k8s.io/kubernetes/pkg/version/verflag"
Code change:
When I tried to use the following command to build kubernetes: I encountered the following errors: I'm using Ubuntu 14.04 and the code from master branch. will submit a patch then to fix this problem.
whoa, I just landed in the exact same place, I am trying to build k8s on a CentOS7 box. Tried with removing the unused utils and works like a charm !

Related discussion:
text: "github.com/spf13/pflag" "k8s.io/kubernetes/contrib/mesos/pkg/executor/service" "k8s.io/kubernetes/contrib/mesos/pkg/hyperkube" <del> "k8s.io/kubernetes/pkg/util" </del> "k8s.io/kubernetes/pkg/util/flag" "k8s.io/kubernetes/pkg/util/logs" "k8s.io/kubernetes/pkg/version/verflag"
Code change:
When I tried to use the following command to build kubernetes: I encountered the following errors: I'm using Ubuntu 14.04 and the code from master branch. will submit a patch then to fix this problem.
whoa, I just landed in the exact same place, I am trying to build k8s on a CentOS7 box. Tried with removing the unused utils and works like a charm !

Related discussion:
text: "github.com/spf13/pflag" "k8s.io/kubernetes/contrib/mesos/pkg/hyperkube" "k8s.io/kubernetes/contrib/mesos/pkg/scheduler/service" <del> "k8s.io/kubernetes/pkg/util" </del> "k8s.io/kubernetes/pkg/util/flag" "k8s.io/kubernetes/pkg/util/logs" "k8s.io/kubernetes/pkg/version/verflag"
Code change:
liveness probe was introduced in Heapster seems to work correctly so we should either fix it or revert liveness probe.
cc
port seems to be wrong

Related discussion:
text: livenessProbe: httpGet: path: /healthz <del> port: 8080 </del> <ins> port: 8082 </ins> scheme: HTTP initialDelaySeconds: 180 timeoutSeconds: 5
Code change:
introduced a new requirement for node names to be DNS1123Labels. In case of hack/local-up-, an IP address is used as node name and thus endpoints aren't created for services, e.g. kube-dns doesn't work anymore. There are other projects that use IP addresses as node names for running simplified k8s dev environment: Probably there are other projects where it's convenient to have just an IP address as a node name. Verified against . On affected installs, service latency e2e test hangs, see e.g. Example ( some newlines):

Related discussion:
text: if len(address.Hostname) > 0 { allErrs = append(allErrs, ValidateDNS1123Label(address.Hostname, fldPath.Child("hostname"))...) } <del> // During endpoint update, validate NodeName is DNS1123 compliant and transition rules allow the update </del> <ins> // During endpoint update, verify that NodeName is a DNS subdomain and transition rules allow the update </ins> if address.NodeName != nil { <del> allErrs = append(allErrs, ValidateDNS1123Label(*address.NodeName, fldPath.Child("nodeName"))...) </del> <ins> for _, msg := range ValidateNodeName(*address.NodeName, false) { allErrs = append(allErrs, field.Invalid(fldPath.Child("nodeName"), *address.NodeName, msg)) } </ins> } allErrs = append(allErrs, validateEpAddrNodeNameTransition(address, ipToNodeName, fldPath.Child("nodeName"))...) if len(allErrs) > 0 {
Code change:
introduced a new requirement for node names to be DNS1123Labels. In case of hack/local-up-, an IP address is used as node name and thus endpoints aren't created for services, e.g. kube-dns doesn't work anymore. There are other projects that use IP addresses as node names for running simplified k8s dev environment: Probably there are other projects where it's convenient to have just an IP address as a node name. Verified against . On affected installs, service latency e2e test hangs, see e.g. Example ( some newlines):

Related discussion:
text: } } <del> func TestEndpointAddressNodeNameInvalidDNS1123(t *testing.T) { </del> <ins> func TestEndpointAddressNodeNameInvalidDNSSubdomain(t *testing.T) { </ins> // Check NodeName DNS validation <del> endpoint := newNodeNameEndpoint("illegal.nodename") </del> <ins> endpoint := newNodeNameEndpoint("illegal*.nodename") </ins> errList := ValidateEndpoints(endpoint) if len(errList) == 0 { t.Error("Endpoint should reject invalid NodeName") } } <ins> func TestEndpointAddressNodeNameCanBeAnIPAddress(t *testing.T) { endpoint := newNodeNameEndpoint("10.10.1.1") errList := ValidateEndpoints(endpoint) if len(errList) != 0 { t.Error("Endpoint should accept a NodeName that is an IP address") } } </ins>
Code change:
During a manual upgrade test, described in , a kubelet failed to communicate via SSL to its master. This is currently blocking the 1.4 release. There is not a repro for this yet. The cluster was running 1.3, and was upgraded to 1.4 successfully, but upon downgrading to 1.3, the kubelets one one or more machines are not able to talk to the master. Current kubelet version is . (EDITED) The error when talking to the master is (lots like this): I think this means that the kubelet cannot verify the master's cert. I can't find the master's cert. Since there is not flag to kubelet to tell it where to find its config, I assume it is using the default location, which is . It seems wrong that clusters[0].cluster.certificate-authority-data is empty. That should have the cert for the master, right? I see this thing: but I don't understand what process generates , and if, presumably, from that template, why it isn't setting the env var CA_CERT.
in case this is related to kubelet bootstrap changes in 1.4.
Currently node is :
Correction. The node is running 1.4.0-beta.5
If the kubelet hasn't set the then it will not go through the bootstrap code path. Is the kubeconfig provided through the bootstrap flag or the kubeconfig flag?
Not set. For reference, args are:
Question: Could you please tell me why your nodes are running with GCI image at the first place? The test is starting with 1.3 release, which is using old debian-based containervm image, and I thought doesn't change the image at all (separate topic).
It does change to gci on gce with the default upgrade scripts, i brought this up on an email thread. I'm guessing something was dropped when we translated salt grains (containervm) to gci env vars.
What the default upgrade scripts you were referring to? I just double-checked with test cluster which was upgrade to 1.4 through , the node is stay with debian-based containervm, but Kubernetes version is bumped up to 1.4-beta.X
Ok, I am assigning this to to confirm. If upgrade logic from 1.3 to 1.4 changes the underneath os, that is wrong. I strongly believe this is just a simple operational error.
My understanding is that ignores base images altogether. , Dawn Chen wrote:
Hmm
I've tried and the node image did get updated to GCI. Before upgrade to v1.4: After upgrade to v1.4: After downgrade to v1.3: /cc
helped me debug this. He found that the metadata for the instance did not include the CA_CERT in it. That seems wrong.
Renamed from "x509 failure for kubelet after upgrade test"
Yeah, that's the error you'd get if the proper ca isn't loaded.
Will we get this problem if we don't upgrade to GCI (i.e. we stay with CVM) in -N? I guess we can try it and find out.
Manual upgrades have been successful if we don't switch image types. , David Oppenheimer < :
re: If you don't change the underneath image during the upgrade, the issue shouldn't occur.

Related discussion:
text: node_upgrade=true node_prereqs=false local_binaries=false <ins> env_os_distro=false </ins> while getopts ":MNPlh" opt; do case ${opt} in
Code change:
During a manual upgrade test, described in , a kubelet failed to communicate via SSL to its master. This is currently blocking the 1.4 release. There is not a repro for this yet. The cluster was running 1.3, and was upgraded to 1.4 successfully, but upon downgrading to 1.3, the kubelets one one or more machines are not able to talk to the master. Current kubelet version is . (EDITED) The error when talking to the master is (lots like this): I think this means that the kubelet cannot verify the master's cert. I can't find the master's cert. Since there is not flag to kubelet to tell it where to find its config, I assume it is using the default location, which is . It seems wrong that clusters[0].cluster.certificate-authority-data is empty. That should have the cert for the master, right? I see this thing: but I don't understand what process generates , and if, presumably, from that template, why it isn't setting the env var CA_CERT.
in case this is related to kubelet bootstrap changes in 1.4.
Currently node is :
Correction. The node is running 1.4.0-beta.5
If the kubelet hasn't set the then it will not go through the bootstrap code path. Is the kubeconfig provided through the bootstrap flag or the kubeconfig flag?
Not set. For reference, args are:
Question: Could you please tell me why your nodes are running with GCI image at the first place? The test is starting with 1.3 release, which is using old debian-based containervm image, and I thought doesn't change the image at all (separate topic).
It does change to gci on gce with the default upgrade scripts, i brought this up on an email thread. I'm guessing something was dropped when we translated salt grains (containervm) to gci env vars.
What the default upgrade scripts you were referring to? I just double-checked with test cluster which was upgrade to 1.4 through , the node is stay with debian-based containervm, but Kubernetes version is bumped up to 1.4-beta.X
Ok, I am assigning this to to confirm. If upgrade logic from 1.3 to 1.4 changes the underneath os, that is wrong. I strongly believe this is just a simple operational error.
My understanding is that ignores base images altogether. , Dawn Chen wrote:
Hmm
I've tried and the node image did get updated to GCI. Before upgrade to v1.4: After upgrade to v1.4: After downgrade to v1.3: /cc
helped me debug this. He found that the metadata for the instance did not include the CA_CERT in it. That seems wrong.
Renamed from "x509 failure for kubelet after upgrade test"
Yeah, that's the error you'd get if the proper ca isn't loaded.
Will we get this problem if we don't upgrade to GCI (i.e. we stay with CVM) in -N? I guess we can try it and find out.
Manual upgrades have been successful if we don't switch image types. , David Oppenheimer < :
re: If you don't change the underneath image during the upgrade, the issue shouldn't occur.

Related discussion:
text: l) local_binaries=true ;; <ins> o) env_os_distro=true ;; </ins> h) usage exit 0
Code change:
./hack/e2e- line 1192: local: `replica-pd=jenkins-us-central1-b-master-pd': not a valid identifier FYI - this looks related to your recent work? I don't see any obvious PR to blame, but I seem to remember you guys touching this part of the code recently? BTW, I deleted a bunch of leaked resources in the relevant test project on Friday (9/16), assuming that the test scripts would re-create whatever they needed. It's possible (but unlikely) that this has anything to do with that.
Here are some example logs to aid debugging: https://k8s-
I believe this to be the source of dysfunction: Which is turn caused by what you found, which causes the disk to not be cleaned up ahead of time: You can try this out in your local shell: git blames leads us to this recent commit: cc the author The root problem is unfortunatley very annoying: this is how bash tells you that variable names (in this case, $replica-pd) cannot contain dashes!
fix is up in
Awesome many thanks. I'll make sure it gets merged ASAP.

Related discussion:
text: # Delete the master replica pd (possibly leaked by kube-up if master create failed). # TODO(jszczepkowski): remove also possibly leaked replicas' pds <del> local -r replica-pd="${REPLICA_NAME:-${MASTER_NAME}}-pd" if gcloud compute disks describe "${replica-pd}" --zone "${ZONE}" --project "${PROJECT}" &>/dev/null; then </del> <ins> local -r replica_pd="${REPLICA_NAME:-${MASTER_NAME}}-pd" if gcloud compute disks describe "${replica_pd}" --zone "${ZONE}" --project "${PROJECT}" &>/dev/null; then </ins> gcloud compute disks delete  --project "${PROJECT}"  --quiet  --zone "${ZONE}"  <del> "${replica-pd}" </del> <ins> "${replica_pd}" </ins> fi # Delete disk for cluster registry if enabled
Code change:
After PR , I did a bit experiments on kube-dns graceful termination and got some unintended result as below: I guess it is because the other container was terminated before . DNS query from clients failed because is the actual connection point. After a bit discussion with and two solutions for this are: another program wrap up the dnsmasq and capture the SIGTERM signal. out a way to modify the sig mask of process without touching the original codes.
Something I have been meaning to try: Put in Dockerfile Build and push image We should deliver SIGCONT rather than SIGTERM. I did a quick test and it seems to work. Otherwise trapping SIGTERM in the shell and ignoring it is well documented... , Zihong Zheng wrote:
Thanks, I will take a look and test these.

Related discussion:
text: <ins> # This Dockerfile will build an image that is configured # to run Fluentd with an Elasticsearch plug-in and the # provided configuration file. # TODO(satnam6502): Use a lighter base image, e.g. some form of busybox. # The image acts as an executable for the binary /usr/sbin/td-agent. # Note that fluentd is run with root permssion to allow access to # log files with root only access under /var/lib/docker/containers/* # Please see http://docs.fluentd.org/articles/install-by-deb for more # information about installing fluentd using deb package. FROM ubuntu:14.04 MAINTAINER Satnam Singh "satnam@google.com" # Ensure there are enough file descriptors for running Fluentd. RUN ulimit -n 65536 # Install prerequisites. RUN apt-get update &&  apt-get install -y curl &&  apt-get install -y -q libcurl4-openssl-dev make &&  apt-get clean # Install Fluentd. RUN /usr/bin/curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh # Change the default user and group to root. # Needed to allow access to /var/log/docker/... files. RUN sed -i -e "s/USER=td-agent/USER=root/" -e "s/GROUP=td-agent/GROUP=root/" /etc/init.d/td-agent # Install the Elasticsearch Fluentd plug-in. RUN /usr/sbin/td-agent-gem install fluent-plugin-elasticsearch # Copy the Fluentd configuration file. COPY td-agent.conf /etc/td-agent/td-agent.conf # Copy a script that determines the name of the host machine # and then patch the Fluentd configuration files and then # run Fluentd in the foreground. ADD run.sh /run.sh # Always run the this setup script. ENTRYPOINT ["/run.sh"] </ins>
Code change:
After PR , I did a bit experiments on kube-dns graceful termination and got some unintended result as below: I guess it is because the other container was terminated before . DNS query from clients failed because is the actual connection point. After a bit discussion with and two solutions for this are: another program wrap up the dnsmasq and capture the SIGTERM signal. out a way to modify the sig mask of process without touching the original codes.
Something I have been meaning to try: Put in Dockerfile Build and push image We should deliver SIGCONT rather than SIGTERM. I did a quick test and it seems to work. Otherwise trapping SIGTERM in the shell and ignoring it is well documented... , Zihong Zheng wrote:
Thanks, I will take a look and test these.

Related discussion:
text: <ins> #!/bin/bash # Copyright 2014 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Build the fluentd-elasticsearch image and push # to google/fluentd-elasticsearch. sudo docker build -t kubernetes/fluentd-elasticsearch . sudo docker push kubernetes/fluentd-elasticsearch </ins>