jiebi/CodeConvo huggingface dataset:
Change:
As mentioned in when the service is deleted from the underlying cluster, the controller should recreate it, but it does not. Below's a transcript. is talking to the federation-apiserver, and is talking to the normal cluster it's pointed at.
Thanks for catching this cc
cc
This is definitely a bug, but I think that we can kick it out into v1.4. Until fixed (i.e. while in beta), we can tell customers not to delete the underlying services in their clusters.
it should not be, I have tested this before. How long did you wait when you check the service existence in last step? Can you send me the logs? The service deletion will cause federation service update, and then the federation service watcher will catch the update even and call another round of service updates to all clusters. When you delete the service, if the service got ingress ip already? I am thinking if the deletion did not trigger federation service update? So so no update event generated..
I think that the best way forward will be to encode this in a reproducible e2e test, otherwise there are too many variables to debug effectively. It's great to know that you've tested this case, and it works in at least some scenarios.
one possible solution come into my mind, currently we set the service full sync time to 10 minutes , if we change it back to 30s, then at least during the full sync, the deleted service will be re-created. I will send a PR anyway, you can judge if it should be put to v1.3
I think we need to find and fix the underlying problem, not cover it up with a shorter sync period. If we have the correct watch on each service, we will be notified as soon as they are deleted. Why is the watch notification not firing correctly?
Keeping this open until we've verified this fix in e2e tests.
I believe this to be fixed, but we still lack an e2e test to verify conclusively. would you like to take this on along with the other e2e tests that you are busy adding?
ping
plan on doing this for 1.4? or bump to 1.5?
Yes, 1.5 is fine.
switching this to 1.6 as it's too late for 1.5. ok?
Moving to 1.7 as late to happen in 1.6. Feel free to switch back if this is incorrect.

Discussion:
clusterCache.serviceQueue.Add(key) return err } <del> var needUpdate bool </del> <ins> var needUpdate, isDeletion bool </ins> if exists { service, ok := serviceInterface.(*v1.Service) if ok {

} glog.Infof("Found tombstone for %v", key) needUpdate = cc.processServiceDeletion(cachedService, clusterName) <ins> isDeletion = true </ins> } } else { glog.Infof("Can not get service %v for cluster %s from serviceStore", key, clusterName) needUpdate = cc.processServiceDeletion(cachedService, clusterName) <ins> isDeletion = true </ins> } if needUpdate {

} } } <ins> if isDeletion { // cachedService is not reliable here as // deleting cache is the last step of federation service deletion _, err := fedClient.Core().Services(cachedService.lastState.Namespace).Get(cachedService.lastState.Name) // rebuild service if federation service still exists if err == nil || !errors.IsNotFound(err) { return sc.ensureClusterService(cachedService, clusterName, cachedService.appliedState, clusterCache.clientset) } } </ins> return nil }
Change:
I want to isolate these in their own project:
Right now just the CI jobs, but after we also need to do this for the PR jobs.

Discussion:
ARTIFACTS=${WORKSPACE}/_artifacts mkdir -p ${ARTIFACTS} <ins> if [[ -f "${KUBEKINS_SERVICE_ACCOUNT_FILE:-}" ]]; then echo 'Activating service account...'  # No harm in doing this multiple times. gcloud auth activate-service-account --key-file="${KUBEKINS_SERVICE_ACCOUNT_FILE}" # https://developers.google.com/identity/protocols/application-default-credentials export GOOGLE_APPLICATION_CREDENTIALS="${KUBEKINS_SERVICE_ACCOUNT_FILE}" unset KUBEKINS_SERVICE_ACCOUNT_FILE elif [[ -n "${KUBEKINS_SERVICE_ACCOUNT_FILE:-}" ]]; then echo "ERROR: cannot access service account file at: ${KUBEKINS_SERVICE_ACCOUNT_FILE}" fi </ins> go run test/e2e_node/runner/run_e2e.go  --logtostderr --vmodule=*=4 --ssh-env="gce"  --zone="$GCE_ZONE" --project="$GCE_PROJECT" --hosts="$GCE_HOSTS"  --images="$GCE_IMAGES" --image-project="$GCE_IMAGE_PROJECT" 

GCE_HOSTS= GCE_IMAGE_CONFIG_PATH=test/e2e_node/jenkins/image-config.yaml GCE_ZONE=us-central1-f <del> GCE_PROJECT=kubernetes-jenkins-pull </del> <ins> GCE_PROJECT=k8s-jkns-pr-node-e2e </ins> CLEANUP=true GINKGO_FLAGS='--skip="[Flaky]|[Slow]|[Serial]"' SETUP_NODE=false
Change:
I want to isolate these in their own project:
Right now just the CI jobs, but after we also need to do this for the PR jobs.

Discussion:
GCE_HOSTS= GCE_IMAGE_CONFIG_PATH=test/e2e_node/jenkins/image-config.yaml GCE_ZONE=us-central1-f <del> GCE_PROJECT=kubernetes-jenkins </del> <ins> GCE_PROJECT=k8s-jkns-ci-node-e2e </ins> CLEANUP=true GINKGO_FLAGS='--skip="[Flaky]|[Serial]"' SETUP_NODE=false

GCE_IMAGES=${GCI_IMAGE} GCE_IMAGE_PROJECT=${GCI_IMAGE_PROJECT} GCE_ZONE=us-central1-f <del> GCE_PROJECT=kubernetes-jenkins </del> <ins> GCE_PROJECT=k8s-jkns-ci-node-e2e </ins> # user-data is the GCI cloud init config file. # gci-docker-version specifies docker version in GCI image. GCE_INSTANCE_METADATA="user-data<${GCI_CLOUD_INIT},gci-docker-version=${DOCKER_VERSION}"

GCE_HOSTS= GCE_IMAGE_CONFIG_PATH=test/e2e_node/jenkins/image-config.yaml GCE_ZONE=us-central1-f <del> GCE_PROJECT=kubernetes-jenkins </del> <ins> GCE_PROJECT=k8s-jkns-ci-node-e2e </ins> CLEANUP=true GINKGO_FLAGS='--focus="[Serial]" --skip="[Flaky]"' SETUP_NODE=false
Change:
Current the federation-controller-manager docker images does not include a root ca cert bundle which causes syncing with DNS providers to fail: To fix this issue I created my own container image using the following Dockerfile: This resulted in a working image with a root cert bundle. Syncing with the GCE DNS provider now works.
cc: FYI...
I'm willing to grab this tomorrow.
Cool, thanks . Re-assigning.

Discussion:
app: federated-cluster module: federation-controller-manager spec: <ins> volumes: - name: ssl-certs hostPath: path: /etc/ssl/certs </ins> containers: - name: controller-manager <ins> volumeMounts: - name: ssl-certs readOnly: true mountPath: /etc/ssl/certs </ins> image: {{.FEDERATION_CONTROLLER_MANAGER_IMAGE_REPO}}:{{.FEDERATION_CONTROLLER_MANAGER_IMAGE_TAG}} command: - /usr/local/bin/federation-controller-manager
Change:
Only a single pod per node is able to use the same Ceph RBD image in read-only mode. Further pods fail to be scheduled with . I would expect multiple pods per node to be able to use the same RBD image in read-only mode, for example by mapping the image once per node and bind-mounting it into each pod. Is there any reason why this isn't possible?
Will wait for confirmation whether this is as per-design or a bug from a storage area engineer. If this is a bug, need a pointer to the log snippets for further debugging. cc
According to , Ceph RBD forbids any two pods from sharing the same monitor, and match pool and image. for details CC
yes, it is a valid use case. let me run some tests first.
This is an issue with the PV scheduler constraints but works in volumes as mentioned here:
opened a fix.
Great, thanks

Discussion:
if volume.RBD != nil && existingVolume.RBD != nil { mon, pool, image := volume.RBD.CephMonitors, volume.RBD.RBDPool, volume.RBD.RBDImage emon, epool, eimage := existingVolume.RBD.CephMonitors, existingVolume.RBD.RBDPool, existingVolume.RBD.RBDImage <del> if haveSame(mon, emon) && pool == epool && image == eimage { </del> <ins> // two RBDs images are the same if they share the same Ceph monitor, are in the same RADOS Pool, and have the same image name // only one read-write mount is permitted for the same RBD image. // same RBD image mounted by multiple Pods conflicts unless all Pods mount the image read-only if haveSame(mon, emon) && pool == epool && image == eimage && !(volume.RBD.ReadOnly && existingVolume.RBD.ReadOnly) { </ins> return true } }
Change:
TestUpdateNewNodeStatus TestUpdateExistingNodeStatus TestUpdateNodeStatusWithRuntimeStateError The above tests seem to be architecture specific now (and fail on unexpected objects "linux" vs "darwin"). Per I would expect unit tests to still pass on OSX

Discussion:
"net/http" "os" "reflect" <ins> goruntime "runtime" </ins> "sort" "strconv" "strings"

BootID:                  "1b3", KernelVersion:           "3.16.0-0.bpo.4-amd64", OSImage:                 "Debian GNU/Linux 7 (wheezy)", <del> OperatingSystem:         "linux", Architecture:            "amd64", </del> <ins> OperatingSystem:         goruntime.GOOS, Architecture:            goruntime.GOARCH, </ins> ContainerRuntimeVersion: "test://1.5.0", KubeletVersion:          version.Get().String(), KubeProxyVersion:        version.Get().String(),
Change:
Run like so:
nodeStore doesnt seem to be set anywhere (except in tests). Thats why code panics when we access it:
cc
Sent to fix this
thx

Discussion:
domain:           domain, cache:            NewTreeCache(), cacheLock:        sync.RWMutex{}, <ins> nodesStore:       kcache.NewStore(kcache.MetaNamespaceKeyFunc), </ins> reverseRecordMap: make(map[string]*skymsg.Service), domainPath:       reverseArray(strings.Split(strings.TrimRight(domain, "."), ".")), federations:      federations,
Change:
Federated service shards are created only in the clusters that have the specified namespace. To fix this problem we need to create the namespace on-demand while creating the federated service shards. cc
As discussed, we explicitly decided not to do it this way. In Kubernetes, when one creates an entity, if the namespace does not exist, the creation call fails. What we need to do for v1.3 is pre-create the test namespace in all of the underlying clusters as part of the e2e test turnup.

Discussion:
"k8s.io/kubernetes/federation/apis/federation" "k8s.io/kubernetes/federation/client/clientset_generated/federation_release_1_3" "k8s.io/kubernetes/pkg/api" <ins> "k8s.io/kubernetes/pkg/api/errors" </ins> "k8s.io/kubernetes/pkg/api/v1" "k8s.io/kubernetes/pkg/client/clientset_generated/release_1_3" "k8s.io/kubernetes/pkg/client/restclient"

clset := release_1_3.NewForConfigOrDie(restclient.AddUserAgent(cfg, UserAgentName)) clusterClientSets = append(clusterClientSets, clset) } <ins> for i, cs := range clusterClientSets { if _, err := cs.Core().Namespaces().Get(f.Namespace.Name); errors.IsNotFound(err) { ns := &v1.Namespace{ ObjectMeta: v1.ObjectMeta{ Name: f.Namespace.Name, }, } if _, err := cs.Core().Namespaces().Create(ns); err != nil { framework.Logf("Couldn't create the namespace %s in cluster [%d]: %v", f.Namespace.Name, i, err) } framework.Logf("Namespace %s created in cluster [%d]", f.Namespace.Name, i) } else if err != nil { framework.Logf("Couldn't create the namespace %s in cluster [%d]: %v", f.Namespace.Name, i, err) } } </ins> }) AfterEach(func() { framework.SkipUnlessFederated(f.Client) <ins> for i, cs := range clusterClientSets { if err := cs.Core().Namespaces().Delete(f.Namespace.Name, &api.DeleteOptions{}); err != nil { framework.Failf("Couldn't delete the namespace %s in cluster [%d]: %v", f.Namespace.Name, i, err) } framework.Logf("Namespace %s deleted in cluster [%d]", f.Namespace.Name, i) } </ins> // Delete the registered clusters in the federation API server. clusterList, err := f.FederationClientset.Federation().Clusters().List(api.ListOptions{}) Expect(err).NotTo(HaveOccurred())
Change:
https://k8s- Failed: [] NodeProblemDetector [] KernelMonitor should generate node condition and events for corresponding errors {Kubernetes e2e suite}

Discussion:
Consistently(func() error { return verifyNoEvents(c.Events(eventNamespace), eventListOptions) }, pollConsistent, pollInterval).Should(Succeed()) <del> By("Make sure the default node condition is false") Consistently(func() error { </del> <ins> By("Make sure the default node condition is generated") Eventually(func() error { </ins> return verifyCondition(c.Nodes(), node.Name, condition, api.ConditionFalse, defaultReason, defaultMessage) }, pollConsistent, pollInterval).Should(Succeed())
Change:
https://k8s- Failed: [] Pods should have their auto-restart back-off timer reset on image update [Slow] {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Pods should have their auto-restart back-off timer reset on image update [Slow] {Kubernetes e2e suite}
https://k8s- Failed: [] Pods should have their auto-restart back-off timer reset on image update [Slow] {Kubernetes e2e suite}
https://k8s- Failed: [] Pods should have their auto-restart back-off timer reset on image update [Slow] {Kubernetes e2e suite}
https://k8s- Failed: [] Pods should have their auto-restart back-off timer reset on image update [Slow] {Kubernetes e2e suite}

Discussion:
Consistently(func() error { return verifyNoEvents(c.Events(eventNamespace), eventListOptions) }, pollConsistent, pollInterval).Should(Succeed()) <del> By("Make sure the default node condition is false") Consistently(func() error { </del> <ins> By("Make sure the default node condition is generated") Eventually(func() error { </ins> return verifyCondition(c.Nodes(), node.Name, condition, api.ConditionFalse, defaultReason, defaultMessage) }, pollConsistent, pollInterval).Should(Succeed())
Change:
The path to the server tarball resolves to: is hardcoded, and does not exist in release builds (v1.3.0-beta.2) The file in question is located at: Correcting the path is a workaround for release builds
cc
Hey, sorry about that. I think I got a fix, but I need your help to test it. I'm traveling and my cellular upload speeds are not so great. Please take a look at this and let me know if it solves the issue:

Discussion:
function upload-resources() { swift post kubernetes --read-acl '.r:*,.rlistings' <del> echo "[INFO] Upload ${KUBERNETES_RELEASE_TAR}" swift upload kubernetes ${ROOT}/../../_output/release-tars/${KUBERNETES_RELEASE_TAR}  </del> <ins> locations=( "${ROOT}/../../_output/release-tars/${KUBERNETES_RELEASE_TAR}" "${ROOT}/../../server/${KUBERNETES_RELEASE_TAR}" ) RELEASE_TAR_LOCATION=$( (ls -t "${locations[@]}" 2>/dev/null || true) | head -1 ) RELEASE_TAR_PATH=$(dirname ${RELEASE_TAR_LOCATION}) echo "[INFO] Uploading ${KUBERNETES_RELEASE_TAR}" swift upload kubernetes ${RELEASE_TAR_PATH}/${KUBERNETES_RELEASE_TAR}  </ins> --object-name kubernetes-server.tar.gz <del> echo "[INFO] Upload kubernetes-salt.tar.gz" swift upload kubernetes ${ROOT}/../../_output/release-tars/kubernetes-salt.tar.gz  </del> <ins> echo "[INFO] Uploading kubernetes-salt.tar.gz" swift upload kubernetes ${RELEASE_TAR_PATH}/kubernetes-salt.tar.gz  </ins> --object-name kubernetes-salt.tar.gz }
Change:
https://k8s- Failed: [] Container Runtime Conformance Test container runtime conformance blackbox test when running a container with a new image should not be able to pull non-existing image from {E2eNode Suite}
https://k8s- Failed: [] Container Runtime Conformance Test container runtime conformance blackbox test when running a container with a new image should not be able to pull non-existing image from {E2eNode Suite}
https://k8s- Failed: [] Container Runtime Conformance Test container runtime conformance blackbox test when running a container with a new image should not be able to pull non-existing image from {E2eNode Suite}
The fix has been merged , close this one. Feel free to reopen if this happens again.

Discussion:
Expect(container.Create()).To(Succeed()) defer container.Delete() <del> By("check the pod phase") Eventually(container.GetPhase, retryTimeout, pollInterval).Should(Equal(testCase.phase)) Consistently(container.GetPhase, consistentCheckTimeout, pollInterval).Should(Equal(testCase.phase)) </del> <ins> // We need to check container state first. The default pod status is pending, If we check // pod phase first, and the expected pod phase is Pending, the container status may not // even show up when we check it. </ins> By("check the container state") <del> status, err := container.GetStatus() Expect(err).NotTo(HaveOccurred()) Expect(GetContainerState(status.State)).To(Equal(testCase.state)) </del> <ins> getState := func() (ContainerState, error) { status, err := container.GetStatus() if err != nil { return ContainerStateUnknown, err } return GetContainerState(status.State), nil } Eventually(getState, retryTimeout, pollInterval).Should(Equal(testCase.state)) Consistently(getState, consistentCheckTimeout, pollInterval).Should(Equal(testCase.state)) By("check the pod phase") Expect(container.GetPhase()).To(Equal(testCase.phase)) </ins> By("it should be possible to delete") Expect(container.Delete()).To(Succeed())
Change:
Because of the revert on commit , when building for 3rd party registry, hyperkube is broken again.
Can we close this now? looks like the PR has merged

Discussion:
"containers": [ { "name": "kube-proxy", <del> "image": "gcr.io/google_containers/hyperkube-ARCH:VERSION", </del> <ins> "image": "REGISTRY/hyperkube-ARCH:VERSION", </ins> "command": [ "/hyperkube", "proxy",
Change:
Kubernetes is failing to start on AWS wily. Following the guide from here: Here's a snippet from /var/log/syslog
More logs,
cc
Is there a workaround fix I can use in the scripts to get around this?

Discussion:
- mode: 644 - makedirs: true <ins> libltdl7: pkg.installed </ins> docker-upgrade: cmd.run: - name: /opt/kubernetes/helpers/pkg install-no-start {{ docker_pkg_name }} {{ override_docker_ver }} /var/cache/docker-install/{{ override_deb }} - require: - file: /var/cache/docker-install/{{ override_deb }} <ins> - pkg: libltdl7 </ins> {% endif %} # end override_docker_ver != ''
Change:
The command was removed from the dockerfile in this PR: This addition was originally proposed here: I'd like to fix the problem or re-enable the creation of symlinks if none exists. Currently there is not enough information from the PR that removed them to know what the problem is. I've tested the symlink command and successfully have built hyperkube images with it enabled. Calling the kubelet symlink doesn't panic for me. Do you know how to reproduce this issue?
You're right, the panic I observed when giving no arguments is gone with v1.3.0 It doesn't work when cross-building Qemu doesn't play very well with Go binaries Can't we just for all binaries in the hyperkube image? I can make a patch with that if you like
It was discussed in that the idea with the command was there to obviate the need to maintain all the commands in the dockerfile since the hyperkube binary already knows all the registered servers. Do you have more information about why this doesn't work with cross-building? I don't quite understand why Go binaries not playing nicely with qemu matter, since, at some point hyperkube is full of a bunch of go binaries that have to play nicely with arm.
Qemu can't run Go binaries, that's the problem. We can use virtually anywhere else Until Qemu releases a version that's compatible with Go, we can't run Go binaries when cross-building
Ooh I understand now, basically any use of go during cross building will not work. Do you happen to know if this more a Go issue or qemu issue? Which upstream project needs to make a fix?
It's a qemu issue, talked to the golang folks Some qemu maintainer said they've improved stuff at HEAD, but haven't tested it Anyway, we have to wait until the next qemu release. If we want it now, we initially have to go with

Discussion:
COPY cni-conf /etc/cni/net.d # Create symlinks for each hyperkube server <del> # TODO: this is unreliable for now (e.g. running "/kubelet" panics) # Also, it doesn't work for other architectures </del> <ins> # TODO: replace manual symlink creation with --make-symlink command once # cross-building with qemu supports go binaries. See #28702 </ins> # RUN /hyperkube --make-symlinks <ins> RUN ln -s /hyperkube /apiserver  && ln -s /hyperkube /controller-manager  && ln -s /hyperkube /federation-apiserver  && ln -s /hyperkube /federation-controller-manager  && ln -s /hyperkube /kubectl  && ln -s /hyperkube /kubelet  && ln -s /hyperkube /proxy  && ln -s /hyperkube /scheduler </ins>
Change:
It should be possible to pass empty environment variables to pods run by env (to clear an env var set inside the image, for example). is valid syntax for "set env var FOO to empty string".

Discussion:
} name := env[:pos] value := env[pos+1:] <del> if len(name) == 0 || len(value) == 0 { </del> <ins> if len(name) == 0 { </ins> return nil, fmt.Errorf("invalid env: %v", env) } if len(validation.IsCIdentifier(name)) != 0 {

} name := env[:pos] value := env[pos+1:] <del> if len(name) == 0 || len(validation.IsCIdentifier(name)) != 0 || len(value) == 0 { </del> <ins> if len(name) == 0 || len(validation.IsCIdentifier(name)) != 0 { </ins> return nil, fmt.Errorf("invalid env: %v", env) } envVar := v1.EnvVar{Name: name, Value: value}

envArray: []string{ "WITH_OUT_VALUES=", }, <del> expected:  []api.EnvVar{}, expectErr: true, </del> <ins> expected: []api.EnvVar{ { Name:  "WITH_OUT_VALUES", Value: "", }, }, expectErr: false, </ins> test:      "test case 3", }, {
Change:
abac-authz- hardcodes "admin" instead of using the $KUBE_USER variable. This means that "admin" is the only basic-auth username that will work with that abac config.
To workaround until it is fixed, you can add a line to the master's abac-authz- file: and then restart the kube-apiserver (docker kill it, and let it restart).

Discussion:
fi env-to-grains "runtime_config" <ins> env-to-grains "kube_user" </ins> } function salt-node-role() {

webhook_config_volume="{"name": "webhookconfigmount","hostPath": {"path": "/etc/gcp_authz.config"}}," fi local -r src_dir="${KUBE_HOME}/kube-manifests/kubernetes/gci-trusty" <del> cp "${src_dir}/abac-authz-policy.jsonl" /etc/srv/kubernetes/ </del> <ins> local -r abac_policy_json="${src_dir}/abac-authz-policy.jsonl" remove-salt-config-comments "${abac_policy_json}" sed -i -e "s@{{kube_user}}@${KUBE_USER}@g" "${abac_policy_json}" cp "${abac_policy_json}" /etc/srv/kubernetes/ </ins> src_file="${src_dir}/kube-apiserver.manifest" remove-salt-config-comments "${src_file}" # Evaluate variables.

fi src_dir="/home/kubernetes/kube-manifests/kubernetes/gci-trusty" <del> cp "${src_dir}/abac-authz-policy.jsonl" /etc/srv/kubernetes/ </del> <ins> local -r abac_policy_json="${src_dir}/abac-authz-policy.jsonl" remove_salt_config_comments "${abac_policy_json}" sed -i -e "s@{{kube_user}}@${KUBE_USER}@g" "${abac_policy_json}" cp "${abac_policy_json}" /etc/srv/kubernetes/ </ins> src_file="${src_dir}/kube-apiserver.manifest" remove_salt_config_comments "${src_file}" # Evaluate variables

<ins> {% set kube_user = grains.kube_user -%} </ins> {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user":"admin", "namespace": "*", "resource": "*", "apiGroup": "*", "nonResourcePath": "*"}} <ins> {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user":"{{kube_user}}", "namespace": "*", "resource": "*", "apiGroup": "*", "nonResourcePath": "*"}} </ins> {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user":"kubelet", "namespace": "*", "resource": "*", "apiGroup": "*", "nonResourcePath": "*"}} {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user":"kube_proxy", "namespace": "*", "resource": "*", "apiGroup": "*", "nonResourcePath": "*"}} {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user":"kubecfg", "namespace": "*", "resource": "*", "apiGroup": "*", "nonResourcePath": "*"}}

/srv/kubernetes/abac-authz-policy.jsonl: file.managed: - source: salt://kube-apiserver/abac-authz-policy.jsonl <ins> - template: jinja </ins> - user: root - group: root - mode: 600