jiebi/CodeConvo huggingface dataset:
Change:
While fixing the issue on describe events for pod, I found kubelet keep restarting monitoring-heapster-controller container: I retrieved the container log, and found it exit due to database creation failure: I didn't follow up on why influxdb issue. cc/
This happens when InfluxDB service is not up. Heapster retries a few times and then eventually fails. The plan is to avoid failing completely and keep retrying DB creation until it succeeds and bubble up appropriate errors. , Dawn Chen wrote:
Influxdb is running happily without issue on the same node:
Related kube-proxy log:
It looks like a kube-proxy issue.
It looks like there are no endpoints for 'monitoring-influxdb' service. The service itself is active and there is a pod that matches the selector of the service. Here is the 'monitoring-influxdb' service: Influxdb pod that matches the selector for 'monitoring-influxdb' service is running. Here are the endpoints from the apiserver: Wonder why there are no endpoints. cc
I ran into a similar issue where endpoints for monitoring-influxdb was not configured. In my case, the pod state took a long time to move to running (even though the containers on the machine were all running fine). But the endpoints were created as soon as the pod changed to state running.
Probed a bit further and it looks like the kubelet is not populating PodIP.
cc/ Can you take a quick look at this one?
FYI: I am still working on this issue.
Here is a little bit more updates: 1) PodIP is missing on all pods running on that particular node; but pods on other nodes with exact same kubelet do have PodIP associated with. 2) docker inspect shows that every pods on bad node have proper "NetworkMode": 3) docker inspect those pod infra containers, they all have proper IP assigned by docker.
The issue is that the kubelet does not ignore old network containers while computing pod status. Working on a fix.
Nice find :) On Mar 26, 2015 5:50 PM, "Vish Kannan" wrote:

Discussion:
if result.err != nil { return nil, err } // Add user container information <del> if dockerContainerName == PodInfraContainerName { </del> <ins> if dockerContainerName == PodInfraContainerName && result.status.State.Running != nil { </ins> // Found network container podStatus.PodIP = result.ip } else {

} } <ins> if podStatus.PodIP == "" { return nil, ErrNoPodInfraContainerInPod } </ins> if len(statuses) == 0 && podStatus.PodIP == "" { return nil, ErrNoContainersInPod }
Change:
now shows NodePhase, NodeCondition, etc. information from NodeStatus. It should also show related events like what we did for pods.

Discussion:
// ValidateEvent makes sure that the event makes sense. func ValidateEvent(event *api.Event) errs.ValidationErrorList { allErrs := errs.ValidationErrorList{} <del> if event.Namespace != event.InvolvedObject.Namespace { </del> <ins> // TODO: There is no namespace required for minion if event.InvolvedObject.Kind != "Node" && event.Namespace != event.InvolvedObject.Namespace { </ins> allErrs = append(allErrs, errs.NewFieldInvalid("involvedObject.namespace", event.InvolvedObject.Namespace, "namespace does not match involvedObject")) } if !util.IsDNS1123Subdomain(event.Namespace) {

"github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" "github.com/GoogleCloudPlatform/kubernetes/pkg/labels" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/types" </ins> "github.com/golang/glog" )

pods = append(pods, pod) } <del> events, _ := d.Events(namespace).Search(node) </del> <ins> var events *api.EventList if ref, err := api.GetReference(node); err != nil { glog.Errorf("Unable to construct reference to '%#v': %v", node, err) } else { // TODO: We haven't decided the namespace for Node object yet. ref.UID = types.UID(ref.Name) events, _ = d.Events("").Search(ref) } </ins> return describeNode(node, pods, events) }

// Make an event that kubelet restarted. // TODO: get the real minion object of ourself, // and use the real minion name and UID. <ins> // TODO: what is namespace for node? </ins> ref := &api.ObjectReference{ <del> Kind:      "Minion", </del> <ins> Kind:      "Node", </ins> Name:      kl.hostname, UID:       types.UID(kl.hostname), <del> Namespace: api.NamespaceDefault, </del> <ins> Namespace: "", </ins> } kl.recorder.Eventf(ref, "starting", "Starting kubelet.") }
Change:
I suspected this is a regression introduced by . I observed a pod was restarted and stuck at PENDING phase for a long time (Don't know the reason why it is restarted at the first place, but that is a separate issue) ssh to the machine, I found both containers are running happily on the machine: Checked kubelet's log, I found kubelet bailed out on generating PodStatus because the failure of retrieving HostIP: I didn't look into with more detail yet. If no one pick up this tonight, I will take a look tomorrow morning.
cc
That' interesting. kubelet bug comes from GetHostIP() function. It reads cached Node and parse it's addresses. Apparently Nodes does not have any addresses reported in status field. Is that expected? Pod_cache just ignored missing HostIP, so I'll send a PR to have a similar logic in kubelet.
I sync-ed, but still observed the same issue: Here is the error from
The issue here is that NodeStatus.Address is empty. I am closing this one, and filing a separate one for cloud provider.
I'm seeing this problem with v0.15.0. What is the new tracking issue?
at the end I didn't file a separate issue because I couldn't reproduce the issue later. Feel free to file a new issue with detail please.
I'm also seeing this, but don't have a repro case yet unfortunately. what did you mean when you said you sync-ed?
- where are you running your cluster, and how did you create it?
I’m running the Master related process in one Docker container and the Node related process in another Docker container. All the Nodes are connected using OpenVSwitch GRE tunnels. The Nodes register with the Master using this script
I am receiving this same error running master and node on separate coreos images with vagrant on parallels. I am able to register the node, but then health check fails and results in this error.

Discussion:
podStatus.Host = kl.GetHostname() hostIP, err := kl.GetHostIP() if err != nil { <del> return api.PodStatus{}, fmt.Errorf("Cannot get host IP: %v", err) </del> <ins> glog.Errorf("Cannot get host IP: %v", err) } else { podStatus.HostIP = hostIP.String() </ins> } <del> podStatus.HostIP = hostIP.String() </del> return *podStatus, nil }
Change:
This happens with several different objects that I tried creating.
happens with simplest pod I can find:
have fix, but working on test.

Discussion:
glog.V(2).Infof("couldn't find properties for %s, skipping", key) continue } <ins> if details.Type == nil { glog.V(2).Infof("nil details for %s, skipping", key) continue } </ins> fieldType := *details.Type if value == nil { glog.V(2).Infof("Skipping nil field: %s", key)
Change:
Accessing entities without namespaces in the url makes the selflink to look corrupted (has namespace missing): Example - invoking: http://localhost:8080/api/v1beta3/services/ returns entities with this selflink (note the part after namespaces) "selfLink": "/api/v1beta3/namespaces/%7Bnamespaces%7D/services/guestbook"
I thought accessing namespaced objects without the namespace was just disallowed in v1beta3. ?
Oh, probably list is allowed for components that need to watch all objects across all namespaces, such as kube-proxy. In that case, the selfLink looks broken, yes.
- right. I will look into the self link this afternoon to see what's going on. Self assigning.
The api has to allow fetching all objects cross namespaces without specifying namespace, otherwise it will cause multiple requests to the api - per namespace, big overhead for clients.

Discussion:
} } path = strings.Replace(n.itemPath, "{name}", name, 1) <del> if !n.allNamespaces { path = strings.Replace(path, "{"+n.scope.ParamName()+"}", namespace, 1) } </del> <ins> path = strings.Replace(path, "{"+n.scope.ParamName()+"}", namespace, 1) </ins> return path, "", nil }

<ins> /* Copyright 2015 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package apiserver import ( "testing" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/api/meta" "github.com/emicklei/go-restful" ) func TestScopeNamingGenerateLink(t *testing.T) { selfLinker := &setTestSelfLinker{ t:           t, expectedSet: "/api/v1beta3/namespaces/other/services/foo", name:        "foo", namespace:   "other", } s := scopeNaming{ meta.RESTScopeNamespace, selfLinker, "/api/v1beta3/namespaces/{namespace}/services/{name}", true, } service := &api.Service{ ObjectMeta: api.ObjectMeta{ Name:      "foo", Namespace: "other", }, TypeMeta: api.TypeMeta{ Kind: "Service", }, } _, _, err := s.GenerateLink(&restful.Request{}, service) if err != nil { t.Errorf("Unexpected error %v", err) } } </ins>
Change:
I cant bring up the monitoring addons on my local cluster after I turns out that thats because I am using an older version of docker which does not allow "-" in namespaces. I get error on running . suggested that we can change the path to use googlecontainers instead of google-containers since that works for all versions of docker. Using google-containers only works for docker versions 1.5+ Filling this issue to hear what others think about this.
SGTM
Fine with me. I'm working on moving the e2e images, can make this change at same time.
SGTM. Take note Let me know when we definitely decide to do this and I will re-generate the image names for the images that I have created and pushed.
please double and triple check to get everything :) , Satnam Singh wrote:
Thanks for heads-up. I am in the middle of pushing kubernetes/pause and etcd_server to Will make sure use the right path here.
fwiw GoogleCloudPlatform/kubernetes to complement already merged PRs...

Discussion:
KUBE_GCS_MAKE_PUBLIC=y KUBE_GCS_UPLOAD_RELEASE=y KUBE_GCS_RELEASE_BUCKET=kubernetes-release <del> KUBE_GCS_PROJECT=google-containers </del> KUBE_GCS_RELEASE_PREFIX="ci/${LATEST}" KUBE_GCS_LATEST_FILE="ci/latest.txt" KUBE_GCS_LATEST_CONTENTS=${LATEST}

KUBE_GCS_MAKE_PUBLIC=y KUBE_GCS_UPLOAD_RELEASE=y KUBE_GCS_RELEASE_BUCKET=kubernetes-release <del> KUBE_GCS_PROJECT=google-containers </del> KUBE_GCS_RELEASE_PREFIX=release/${KUBE_RELEASE_VERSION} KUBE_GCS_LATEST_FILE="release/latest.txt" KUBE_GCS_LATEST_CONTENTS=${KUBE_RELEASE_VERSION}
Change:
I believe this just requires changing the image name , but I wanted to check that there isn't other work to do before we move it. IIRC, the pause pod is cached on containervm, so we should make sure the tag is also cached. cc
kubernetes/pause is cached in ContainerVM image, should we remove it after host kubernetes/pause?
I am going to do this, since I plan to push etcd_server to today.
I think we still want to cache it in ContainerVM image, just with the tag.

Discussion:
<ins> .PHONY:	build push IMAGE = pause TAG = 0.8.0 build: ./prepare.sh docker build -t gcr.io/google_containers/$(IMAGE):$(TAG) . push:	build gcloud preview docker push gcr.io/google_containers/$(IMAGE):$(TAG) all:	push </ins>
Change:
Forked from downward, this test has been failing since go/k8s-test/kubernetes-e2e-gce/4042, which lines up well with . Assigning to to fix, though a revert may be in order soon (we've had too much noise).
The issue is that I didn't change the networking e2e to set 'Args' instead of 'Command'.
Clients shouldn't be using the internal representation.
hmmmm ... is this really fixed? not working for me atm.
What behaviors are you seeing?
This is a recent run of our Jenkins server:
Re-closing (that was a success I pasted).
This might be the test failure you mentioned over hangout. The detail information can be found at: The issue is fixed this morning. Thanks!

Discussion:
setEntrypointAndCommand(container, &dockerOpts) <ins> glog.V(3).Infof("Container %v/%v/%v: setting entrypoint "%v" and command "%v"", pod.Namespace, pod.Name, container.Name, dockerOpts.Config.Entrypoint, dockerOpts.Config.Cmd) </ins> dockerContainer, err := r.Client.CreateContainer(dockerOpts) if err != nil { if ref != nil {

{ Name:  "webserver", Image: "kubernetes/nettest:1.1", <del> Command: []string{ </del> <ins> Args: []string{ </ins> "-service=" + name, fmt.Sprintf("-peers=%d", peers), "-namespace=" + ns},
Change:
When I run a density test in a large cluster I've noticed the following error in scheduler and kubelet: E0402 13:42:04. 4061 ] apiserver received an error that is not an api.Status: Put http://127.0.0.1:4001/v2/keys/registry/pods/default/my-hostname-density30--d93e-11e4-8f0b-bc30 5be02f04mlp5u?prevExist=false: dial tcp 127.0.0.1:4001: too many open files This could be either because either etcd or apiserver has too low ulimit -n
cc
Since etcd is running now in a pod it has a limit for fd set to (10e6) so I'll see if increasing this for apiserver helps.
etcd is running in a pod since yesterday afternoon. (It was introduced first on friday evening, rolledback on sunday, and reintroduced yesterday afternoon.) Just FYI, to help co-relate timelines..
I increased the limit for apiserver to 10e6 and started counting opened fd using a simple bash loop: datels /proc/4048/fd | wc -l where 4048 is a PID of apiserver. I haven't noticed any more "too many" errors in few consecutive runs of density test on 50 node cluster with 30 pods per node. At the same time I've noticed that number of opened fd reached ~2000. I'm gonna send a PR to increase the limit to something more reasonable.

Discussion:
# do_start() { <ins> # Raise the file descriptor limit - we expect to open a lot of sockets! ulimit -n 65536 </ins> # Return #   0 if daemon has been started #   1 if daemon was already running
Change:
Self-assigning as I work out a reliable fix.
cc
Running into a few issues. On GCE, did you guys block somewhere to wait for etcd to be running before reporting success on kube-up?
No, except there is a race between addson services, dns and etcd container. We introduced a retry logic on those services.
I assume pulling the image is just so fast that you don't run into e2e errors, it looks like you guys block on wait-for-minions that uses glcloud, so I guess when minions are reported in etcd, then you know you are good...

Discussion:
[Service] EnvironmentFile=/etc/sysconfig/kubelet ExecStart=/usr/local/bin/kubelet "$DAEMON_ARGS" <del> Restart=on-failure </del> <ins> Restart=always </ins> [Install] WantedBy=multi-user.target  No newline at end of file

- source: /kubernetes-vagrant/network_closure.sh - require: - pkg: docker-io <ins> - sls: kubelet </ins> - cwd: / - user: root - group: root

{% endif %} {% if grains['cloud'] is defined and grains['cloud'] == 'vagrant' %} - docker <ins> - kubelet </ins> - sdn {% endif %} {% if grains['cloud'] is defined and grains['cloud'] == 'aws' %}

echo "OPTIONS='-b=kbr0 --selinux-enabled ${DOCKER_OPTS}'" >/etc/sysconfig/docker systemctl daemon-reload systemctl start docker <ins> systemctl restart kubelet </ins> } EOF

# verify master has all required daemons echo "Validating master" local machine="master" <del> local -a required_daemon=("salt-master" "salt-minion" "kube-apiserver" "nginx" "kube-controller-manager" "kube-scheduler") </del> <ins> local -a required_daemon=("salt-master" "salt-minion" "kube-apiserver" "nginx" "kube-controller-manager" "kube-scheduler" "kubelet") </ins> local validated="1" until [[ "$validated" == "0" ]]; do validated="0"

) } # Instantiate a kubernetes cluster function kube-up { get-password
Change:
This test seems flaky, possibly the watch is broken-- investigating.
Example failure:
I think the bug happens when a poll happens at exactly the wrong time, which explains why it's flaky and is also a legit bug.
Unfortunately my change didn't fix this. This is because the object can still get and deleted without us seeing these events (when a poll happens at the exact wrong time). I would have expected to have fixed it, though. Maybe we need to reduce that timeout?
Hmm, we should never end up in a situation where a really quick delete from scheduled pods beat the insert into assumed pods. how are we guarding against this without synchronizing the following bits of code? The ttl should be a knob that we twist based on expected network latency of a cloud provider (and latency of a reflector). Lowering it increases the window of erroneous scheduling decisions made during a delayed watch event. The test times out at exactly the ttl though.
yeah that should be sync'd-- I'll send a PR

Discussion:
framework.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { if pod, ok := obj.(*api.Pod); ok { <del> c.modeler.ForgetPod(pod) </del> <ins> c.modeler.LockedAction(func() { c.modeler.ForgetPod(pod) }) </ins> } }, DeleteFunc: func(obj interface{}) { <del> switch t := obj.(type) { case *api.Pod: c.modeler.ForgetPod(t) case cache.DeletedFinalStateUnknown: c.modeler.ForgetPodByKey(t.Key) } </del> <ins> c.modeler.LockedAction(func() { switch t := obj.(type) { case *api.Pod: c.modeler.ForgetPod(t) case cache.DeletedFinalStateUnknown: c.modeler.ForgetPodByKey(t.Key) } }) </ins> }, }, )
Change:
I0407 16:48:52. ] Forwarding from 6000 -6000 --- FAIL: TestForwardPorts (0.02s) 2: Error reading data: read tcp 127.0.0.1:5000: connection reset by peer FAIL coverage: 57.4% of statements FAIL 0.168s https://travis-
cc
Andy, can you take a look at this one?
I've been unable to reproduce locally (Fedora 21 VM) but I will look again. Dawn Chen wrote:

Discussion:
stream := &fakeUpgradeStream{} c.streams[headers.Get(api.PortHeader)] = stream <del> stream.data = c.portData[headers.Get(api.PortHeader)] </del> <ins> // only simulate data on the data stream for now, not the error stream if headers.Get(api.StreamType) == api.StreamTypeData { stream.data = c.portData[headers.Get(api.PortHeader)] } </ins> return stream, nil }
Change:
This looks like a git merge problem (?): I don't see why glusterfs gets the special treatment... is it an error, or is there something funky about glusterfs?
No, you're right. Merge error. , Justin Santa Barbara < :
my bad, thanks!

Discussion:
func(vs *api.VolumeSource, c fuzz.Continue) { // Exactly one of the fields should be set. //FIXME: the fuzz can still end up nil.  What if fuzz allowed me to say that? <del> fuzzOneOf(c, &vs.HostPath, &vs.EmptyDir, &vs.GCEPersistentDisk, &vs.GitRepo, &vs.Secret, &vs.NFS, &vs.ISCSI) </del> fuzzOneOf(c, &vs.HostPath, &vs.EmptyDir, &vs.GCEPersistentDisk, &vs.GitRepo, &vs.Secret, &vs.NFS, &vs.ISCSI, &vs.Glusterfs) }, func(d *api.DNSPolicy, c fuzz.Continue) {
Change:
From Is this comment still valid?
fwiw, map (as in v1beta1) seems better than the array (in v1beta3), as it was possible to get a particular container status via kubectl using name as the index. Example from
The comment is obsolete and should be deleted. For better or worse, the decision was list. It is possible to write templates using lists. See for an example. FWIW, we're planning to change the template syntax: . We could also just pipe the json into jq in a docker container.

Discussion:
PodIP  string `json:"podIP,omitempty" description:"IP address allocated to the pod; routable at least within the cluster; empty if not yet allocated"` // The list has one entry per container in the manifest. Each entry is currently the output <del> // of `docker inspect`. This output format is *not* final and should not be relied // upon. // TODO: Make real decisions about what our info should look like. Re-enable fuzz test // when we have done this. </del> <ins> // of `docker inspect`. </ins> ContainerStatuses []ContainerStatus `json:"containerStatuses,omitempty" description:"list of container statuses"` }
Change:
It's a small problem. Now in "/etc/hosts" of master in vagrant cluster, there are duplicate "127.0.0.1 localhost". Though it doesn't make any error, I think it's better to remove duplicate content.

Discussion:
echo "Adding $minion to hosts file" echo "$ip $minion" >> /etc/hosts fi <del> echo "127.0.0.1 localhost" >> /etc/hosts # enables cmds like 'kubectl get pods' on master. </del> done <ins> echo "127.0.0.1 localhost" >> /etc/hosts # enables cmds like 'kubectl get pods' on master. </ins> # Update salt configuration mkdir -p /etc/salt/minion.d
Change:
Tried to create a service on a namespace 'staging' which doesn't exist on my k8s. Creation succeeded which is odd enough :) Then navigating to the entity also works: GET on /api/v1beta3/namespaces/staging/services/redis-service Navigating to the namespace itself produces 404 'namespaces "staging" not found': /api/v1beta3/namespaces/staging/ Getting all namespaces also doesn't list that namespace which sounds reasonable given the above. I'd expect one of the two: standard approach: a clear error that this namespace doesn't exist when attempting to create an entity on it. preferred imo, but better than nothing: creating silently behind the scenes that namespace cc
Sounds like an error configuring your environment. Is this using a standard provider environment or did you install your own? The kube-Apiserver needs the following flags set for admission_control=NamespaceLifecycle,NamespaceAutoProvision,LimitRanger,ResourceQuota In your case NamespaceAutoProvision must not be running. Sent from my iPhone
Admission control option NamespaceExists does your goal for 1. It is what OpenShift uses. NamespaceAutoProvision does 2. It is used upstream right now, but I think upstream should switch to NamespaceExists and require explicit creation, but that is harder to enable for things like e2e until more work is done. Sent from my iPhone
this is the hack/local-up- I didn't find the admissioncontrol set there at all, perhaps it's a bug in that particular setup?

Discussion:
echo "Starting etcd" kube::etcd::start <ins> # Admission Controllers to invoke prior to persisting objects in cluster ADMISSION_CONTROL=NamespaceLifecycle,NamespaceAutoProvision,LimitRanger,ResourceQuota </ins> APISERVER_LOG=/tmp/kube-apiserver.log sudo -E "${GO_OUT}/kube-apiserver"  --v=${LOG_LEVEL}  <ins> --admission_control="${ADMISSION_CONTROL}"  </ins> --address="${API_HOST}"  --port="${API_PORT}"  --runtime_config=api/v1beta3
Change:
kubectl log stops working for me all the sudden: A little bit digging into the issue, I found the problem is only limited to /v1beta3: Query /v1beta1 Query through v1beta3
cc/
Does replacing "minions" by "nodes" work?
Cool, looks like it should be a simple kubectl fix, I will do it shortly.

Discussion:
readCloser, err := client.RESTClient.Get(). Prefix("proxy"). <del> Resource("minions"). </del> <ins> Resource("nodes"). </ins> Name(pod.Spec.Host). Suffix("containerLogs", namespace, podID, container). Param("follow", strconv.FormatBool(follow)).