Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: Bug Report /kind bug What happened: I created a PVC for a Managed Disk. The Pod which used the PVC fails to start: What you expected to happen: The PV to mount, format and the Pod to start How to reproduce it (as minimally and precisely as possible): Boot a 1.7.2 or higher, apply this Gist: Anything else we need to know?: N/a Environment: Azure Kubernetes version (use ):Cloud provider or hardware configuration**: Azure OS (e.g. from /etc/os-release):Kernel (e.g. ): Install tools:
/sig azure cc: Observation: I didn't expect to see this: . I have not attempted this with Ubuntu workers.
cc:
This is the output of . I would've expected to be trying to mount .
The output of : (Note, at the end I had bumped kubelet up to 6)
cc:
/cc
/assign
Confirmed this on a v1.7.2 Azure cluster.
I've seen similar behavior on v1.7.2 with unmanaged disks.
what type of VM did you use?
yours is a different issue. Your blob disk was not attached to node. Can be a number of things. Disks does not exist, disk of type premium and machine is standard. Or you have no access on disk. Controller logs will tell you what is wrong the disk was attached on LUN 0 and yet resolved to /dev/sda - we have changed the finddiskbylun specifically to work across different vm types (To support F vm type). Can I ask you? does coreOS performs anything specifc/special re scsi interfaces? Does it add more devices? is this consistent against any node of the cluster? can you ls -la /sys/bus/scsi/devices on this node? and ls -la dev/disk/azure/?
The blob disk are attached to the node according to the Azure API and they show up on the node via fdisk/lsscsi. I've seen other cases where the disk is not attached. This isn't one of those cases. I can provide further details if needed.
Thanks - if this coreOs as well then let's keep it here. If not can I ask you to split into a new issue? (The logs you shared says controller never responded flagging volume as attached)
I'll get you the output of the last command ASAP. It repros against any node in my cluster. Tagging to add some Container Linux folks to weigh in on whether that OS does anything special regarding SCSI interfaces.
I'm running Container Linux 1409.7.0. For what it is worth, I managed to get a node to properly handle PVCs again after downgrading kubelet on the assigned node to v1.6.7 and clearing up some lease locks.
that is a great data point. is there a log line that shows the exact mount that is being attempted? cc
Hi , do you know how to set up a k8s cluster based on CoreOS image?
Should get you started but I think it installs Kubernetes 1.6.x
if you'd like to test Tectonic with Kubernetes 1.7.1, use this release candidate: in place of
For now we are looking at '/dev/disk/azure' to detect os/resource disk for excluding during data disk probing. Verified on CoreOS 1409.7.0 on an Azure VM, the dir is not populated by default. Looks like it is missing a udev rule. Could you try the gist: Put it under '/etc/udev/rules.d', and reboot the machine(agent node where disks attached to). After that should be populated. And then can go on verifying whether the PVC works.
Thanks That did the trick. PR for : I'm going to leave this open though - older 16.04 images, older versions of Ubuntu, and other distros (may) not have these udev rules.
Some updates: I've tried that package with v1.7.1, but seems is not included. So I was only able to create a cluster with unmanaged disks. But this issue is not directly related to managed disk. I was able to reproduce by requesting a dynamic unmanaged pvc to my cluster, resulting similar log message: Later I go to that agent node, and apply the udev file mentioned above. After restarting the node, everything comes to normal again: So the key point here is changes to look at in when mounting disk on Azure. But that path was missing on CoreOS. I believe consulting would be the right approach, the CoreOS image/tectonic might need some update as follow-up.
( Note that 1.7.1 is the old Azure disk implementation, so you wouldn't repro anyway. HEAD of master of uses 1.7.2.) But I've confirmed your suspicion is correct per my reply right before yours. :+1: Thanks!
- can you confirm if this solves the issue you are having as well?
Appears to have solved the issue. I had to remove/re-add the nodes and break the leases on the PVCs disks that were in a failed state, but it seems to be recoverable.
Hi, Dynamic PVC is the cause that StatefulSet deployment always fails due to pods stuck forever in the ContainerCreating step. We run terraform provisioned K8S cluster. Storage class defined as Looking into details I can see: On the master I can see: On the minion where pod was scheduled to run I can see The BLOB VHD attached on the worker node as /dev/sdc seems OK: fdisk shows the /dev/sdc 1GB in the list: Block device looks OK too except the fact that /dev/sdc is not initialized and mounted: Volume attachement acknowledged: Problematic mount process starting now:
Hi according to the log "mount: /dev/sda is already mounted", it's due to the udev rules not applied in current coreos. You could apply udev rules according to above comments or try the following k8s fix (build the kubelet and replace the kubelet binary in the agent)
The other issue seems to be that isn't in the hyperkube image possibly? Not sure, but it should probably be a separate Issue. (Note, I'm still leaving this open because this will affect other distros and consideration should be given to implementing the udev logic in Go in Kubernetes rather than adding a hard-to-discover dependency for K8s on Azure.)
is working on containerizing the mount dependencies for in-tree volume plugins, which should eliminate issues like this.
I can confirm that adding udev rules solves the problem.
I also encountered this problem. What is the official fix? Adding the udev rules seems like a temporary solution. Is it something that needs to be fixed on CoreOS or on Kubernetes itself?
Are these udev rules present upstream or were they specifically only slipped into debian/ubuntu images? Are they present in other images already? RHEL/Atomic/CentOS/etc (aka other distros that folks may want to use with Kubernetes on Azure) I personally think it might make sense to write some Go and remove the dependency on the udev rules.
The "66-azure-" is typically installed as part of the Azure agent package, and upstream cloud-init carries a similar rule for Azure. So if you use at all to create packages then it would be to the package, but of course that's not the case with CoreOS. I'll discuss with CoreOS to see if we can add this rule to their Azure images. For now I guess the best option is to pull the rule manually and trigger udev to create the symlinks. The official upstream source for the Azure storage udev rule is in the WALinuxAgent repo:
I have that udev rule, running latest offical tectonic/coreos - in short it does not solve the issue. I see a lot of those kind of log messages. The nodes are essentially rendered useless (cannot attach / detach new volumes on them. I've tried using acs-engine and tectonic-installer to set up k8s clusters on azure, both struggle with the same issues. Sick and tired of Azure implementation being unstable so will probably migrate away from Azure next week.
Hi if you have udev rule on coreos, then it might be other issue, could you provde below info: k8s version logs: the "CONTAINER ID" of "/hyperkube controlle" controller manager logs
CoreOS version: DISTRIBID="Container Linux by CoreOS" DISTRIBRELEASE=1520.6.0 DISTRIBCODENAME="Ladybug" DISTRIBDESCRIPTION="Container Linux by CoreOS 1520.6.0 (Ladybug)" Kubernetes Versions: Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-05T00:40:04Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5+coreos.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-09-25T20:07:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Description of one of the failing pods (not all with PVC fail, but most): Name: brawny-dolphin-smblab--lv678 Namespace: smblab-system Node: smblab-ne-worker-2/10.0.16.4 Start Time: Fri, 20 Oct 2017 14:27:04 +0200 Labels: app=brawny-dolphin-smblab pod-template-hash= Annotations: checksum/config=f379e1b99b000d7d1c21591fe7d0d145b99b62ba69917a30207b2643ff1af612 checksum/postgres=c8b1038876f14a385b84db1591d4b6421f5fb9565d2c7bf482de1db47e45eecc-8b1664183baf3616e6e16024b1a2ddfae4bd320e6c1e3dac339398578099c668 checksum/redis=9b8a435a12cc650c415ced7e86f1632ae9add40568a9253f50e17f079475c297 checksum/secrets=0b101212004969e06ec2f838d1356dd5e767f9eb00c03aa37b453dc9fc888089 {"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"smblab-system","name":"brawny-dolphin-smblab-","uid":"-b591-11e... Status: Running IP: 10.2.0.69 Created By: ReplicaSet/brawny-dolphin-smblab- Controlled By: ReplicaSet/brawny-dolphin-smblab- Containers: brawny-dolphin-smblab: Container ID: docker://1696316dc3d77811e882344bff790e2a51a29f63f222f85b072b65940923a806 Image: gitlab/gitlab-ee:10.0.3-ee.0 Image ID: docker-pullable://gitlab/gitlab-ee Ports: 8105/TCP, 8065/TCP, 8005/TCP, 22/TCP, 9090/TCP Command: /bin/bash -c sed -i "s/environment ({'GITLABROOTPASSWORD' =initialrootpassword }) if initialrootpassword/environment ({'GITLABROOTPASSWORD' =initialrootpassword, 'GITLABSHAREDRUNNERSREGISTRATIONTOKEN' =node['gitlab']['gitlab-rails']['initialsharedrunnersregistrationtoken'] })/g" && exec /assets/wrapper State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 137 Started: Fri, 20 Oct 2017 19:06:45 +0200 Finished: Fri, 20 Oct 2017 19:10:54 +0200 Ready: False Restart Count: 17 Liveness: http-get http://:8005/healthcheck%3Ftoken=healthchecktoken delay=200s timeout=15s period=10s #success=1 #failure=3 Readiness: http-get http://:8005/healthcheck%3Ftoken=healthchecktoken delay=15s timeout=1s period=10s #success=1 #failure=3 Environment: GITLABEXTERNALSCHEME: <set to the key 'external-scheme' of config map 'brawny-dolphin-smblab-gitlab'Optional: false GITLABEXTERNALHOSTNAME: <set to the key 'external-hostname' of config map 'brawny-dolphin-smblab-gitlab'Optional: false GITLABREGISTRYEXTERNALSCHEME: <set to the key 'registry-external-scheme' of config map 'brawny-dolphin-smblab-gitlab'Optional: false GITLABREGISTRYEXTERNALHOSTNAME: <set to the key 'registry-external-hostname' of config map 'brawny-dolphin-smblab-gitlab'Optional: false GITLABMATTERMOSTEXTERNALSCHEME: <set to the key 'mattermost-external-scheme' of config map 'brawny-dolphin-smblab-gitlab'Optional: false GITLABMATTERMOSTEXTERNALHOSTNAME: <set to the key 'mattermost-external-hostname' of config map 'brawny-dolphin-smblab-gitlab'Optional: false POSTGRESUSER: <set to the key 'postgres-user' of config map 'brawny-dolphin-smblab-postgresql'Optional: false POSTGRESPASSWORD: <set to the key 'postgres-password' in secret 'brawny-dolphin-smblab-postgresql'Optional: false POSTGRESDB: <set to the key 'postgres-dbname' of config map 'brawny-dolphin-smblab-postgresql'Optional: false POSTGRESHOST: brawny-dolphin-smblab-postgresql REDISPASSWORD: <set to the key 'redis-password' in secret 'brawny-dolphin-smblab-redis'Optional: false REDISHOST: brawny-dolphin-smblab-redis MATTERMOSTAPPUID: <set to the key 'mattermost-app-uid' of config map 'brawny-dolphin-smblab-gitlab'Optional: false MATTERMOSTAPPSECRET: <set to the key 'mattermost-app-secret' in secret 'brawny-dolphin-smblab-gitlab'Optional: false DEXAPPUID: <set to the key 'dex-app-uid' of config map 'brawny-dolphin-smblab-gitlab'Optional: false DEXAPPSECRET: <set to the key 'dex-app-secret' in secret 'brawny-dolphin-smblab-gitlab'Optional: false GITLABINITIALSHAREDRUNNERSREGISTRATIONTOKEN: <set to the key 'initial-shared-runners-registration-token' in secret 'brawny-dolphin-smblab-gitlab'Optional: false GITLABOMNIBUSCONFIG: <set to the key 'gitlab-omnibus-config' of config map 'brawny-dolphin-smblab-gitlab'Optional: false GITLABPOSTRECONFIGURECODE: <set to the key 'gitlab-post-reconfigure-code' of config map 'brawny-dolphin-smblab-gitlab'Optional: false GITLABPOSTRECONFIGURESCRIPT: <set to the key 'gitlab-post-reconfigure-script' of config map 'brawny-dolphin-smblab-gitlab'Optional: false Mounts: /etc/gitlab from etc (rw) /gitlab-data from data (rw) from brawny-dolphin-smblab-gitlab-token-mwjm5 (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: brawny-dolphin-smblab-gitlab-data ReadOnly: false etc: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: brawny-dolphin-smblab-gitlab-etc ReadOnly: false brawny-dolphin-smblab-gitlab-token-mwjm5: Type: Secret (a volume populated by a Secret) SecretName: brawny-dolphin-smblab-gitlab-token-mwjm5 Optional: false QoS Class: BestEffort Node-Selectors: <noneTolerations: <noneEvents: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 1h (x40 over 3h) kubelet, smblab-ne-worker-2 Unable to mount volumes for pod "brawny-dolphin-smblab--lv678smblab-system(-b591-11e7-8257-)": timeout expired waiting for volumes to attach/mount for pod "smblab-system"/"brawny-dolphin-smblab--lv678". list of unattached/unmounted volumes=[etc] Normal SuccessfulMountVolume 1h kubelet, smblab-ne-worker-2 MountVolume.SetUp succeeded for volume "pvc--b591-11e7-83c7-" Normal Pulled 7m (x18 over 1h) kubelet, smblab-ne-worker-2 Container image "gitlab/gitlab-ee:10.0.3-ee.0" already present on machine Normal Created 7m (x18 over 1h) kubelet, smblab-ne-worker-2 Created container Normal Started 7m (x18 over 1h) kubelet, smblab-ne-worker-2 Started container Warning Unhealthy 3m (x37 over 1h) kubelet, smblab-ne-worker-2 Liveness probe failed: Get http://10.2.0.69:8005/healthcheck?token=healthchecktoken: dial tcp 10.2.0.69:8005: getsockopt: connection refused Warning Unhealthy 2m (x432 over 1h) kubelet, smblab-ne-worker-2 Readiness probe failed: Get http://10.2.0.69:8005/healthcheck?token=healthchecktoken: dial tcp 10.2.0.69:8005: getsockopt: connection refused Normal Killing 2m (x18 over 1h) kubelet, smblab-ne-worker-2 Killing container with id docker://brawny-dolphin-smblab:pod "brawny-dolphin-smblab--lv678smblab-system(-b591-11e7-8257-)" container "brawny-dolphin-smblab" is unhealthy, it will be killed and re-created. Warning FailedSync 12s (x220 over 4h) kubelet, smblab-ne-worker-2 Error syncing pod Warning BackOff 12s (x139 over 1h) kubelet, smblab-ne-worker-2 Back-off restarting failed container Controller manager logs are wiped (terminated the cluster), will provide them if the issue raises again with the new cluster being spawned. It complained about timeouts and azure disk.
thanks for the logs, could you pls also check your to make sure udev rule is applied on your coreos. For your case, I could not make sure it's related to udev rule or not. Two more logs are important for diagnose your issue, thanks. the "CONTAINER ID" of "/hyperkube controlle" controller manager logs And for this udev issue, finally i decided to fix it on k8s side, once this PR is merged into master branch, I will merge into v1.7, v1.8, and according to my analysis, v1.6 won't have this udev issue, so if you are really hitting udev issue, you may try v1.6 instread. Thanks for reporting this, let me know if you have any other issue.
Ok, I encountered an issue with regards to PVC volumes again. Don't know if this is exactly the same, but it is all the situations I have access to atm. Yesterday I deployed a chart (gitlab) which allocates a couple of PVC's, today I upgraded the chart which caused a pod to be terminated and launched on another node. The volumes already attached to the first node then started to move over to the new node. Currently the pod have no logs, it's still in "Waiting: ContainerCreating" state. When logged into the first node (smblab-ne-worker-2, the new node for the given pod is smblab-ne-worker-1) I have this in my folder: When checking the hyperkube controller logs I get this The error get repeated a gazillion of times before this, I pasted the latest relevant logs. It tried to detach to volumes from . When showing the VM information () I get the following: The datadisk attached in this information is supposed to be there and be attached, the two pvcs which was supposed to move is not attached - still provisioningState is updating. So K8S won't start the pod on the new node, it's waiting for the detachmentprocess to complete. Azure Portal says "Updating..." when visiting VM . I can't see the PVC's in either. Don't know if this is exactly the same issue as discussed in this thread, but I guess it is related. Currently I have been waiting over 1 hour for the pod to start, I suspect that it won't. I believe attaching new PVCs to this node is also difficult, as long as it's stuck in this provisioningState. Neither of the two PVCs which was supposed to move from to is attached to the new node. edit Now I noticed that the extension got the in the pasted json, it may be that which cause it to get stuck.
Logs for waagent service ():
Removed that extension from the VM and it progressed into "Succeeded" state, pod now up and running. Notifying CoreOS about the issue.
So everything is ok? If it's OMSAgent issue, you could file a issue in
It looks OK, will try stressing the cluster later today (multiple pvc related requests). Does OmsAgent provide any required features? It apparently "updates" itself which is not what I prefer (I want to know what's running). My guess is that the agent worked when CoreOS built the image, but the update breaks it. I prefer to remove it if it can lock all nodes in "updating" state.
I have no idea what omsagent does, I think you remove it first and file an issue to omsagent project. Just ping me if you have any azure pvc related issue, thx for the try.
thanks folks for the discussion, the fix has been merged into master and I will merge this fix into v1.7, v1.8 v1.7: v1.8:

Discussion:
if len(arr) < 4 { continue } <ins> if len(azureDisks) == 0 { glog.V(4).Infof("/dev/disk/azure is not populated, now try to parse %v directly", name) target, err := strconv.Atoi(arr[0]) if err != nil { glog.Errorf("failed to parse target from %v (%v), err %v", arr[0], name, err) continue } // as observed, targets 0-3 are used by OS disks. Skip them if target <= 3 { continue } } </ins> // extract LUN from the path. // LUN is the last index of the array, i.e. 1 in /sys/bus/scsi/devices/3:0:0:1 l, err := strconv.Atoi(arr[3])