Change:
Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Install tools: default install What happened: AWS EBS backed PersitantVolumeClaim created a volume in an AZ where there was a master but no nodes. Making it unusable. What you expected to happen: Zones with master and no nodes shouldn't be used to create the EBS Volume How to reproduce it (as minimally and precisely as possible): AWS setup across 3 AZs in one region Master in three AZs Nodes in 2 AZs (so one AZ has a Master but no Nodes) Create a Storage class to use EBS without specifying AZCreate PVC against StorageClassPVC can create a volume in an availability zone with no nodes Anything else do we need to know: Mentioned on Slack channel #sig-aws with who asked me to raise this
, Mengqi Yu wrote:
Running Kubernetes master in a zone that has no running nodes is quite strange and I don't think this is a typical case we should support. AWS cloud provider lists all machines in the cluster and chooses a random zone that has at least one running instance. It seems it does not distinguish master/node machines. IMO, GCE has the same problem, it lists all zones in a region where the master runs and it does not distinguish masters and nodes either. We can either introduce a new AWS instance tag with master/node role that is running on the instance or we can move the zone decision into volume plugin itself, where we have a kubeclient and we can filter nodes relatively easily, however we will depend on an admission controller that must add a zone label to nodes. thoughts? As a third option, we could rely on kubernetes admin to fill all zones into StorageClass in this weird case. Right now we support only one zone in "zone" parameter, we could allow a list:
just to add to this, the reason why it's a deal is I actually don't have full control over where the nodes run, as they're in an autoscale group across all AZs. So you could have master in AZ a and nodes in b and c just because that's what the auto scale group picked.
I just ran into this, however, I only have nodes / masters in 3 zones, but the PV was created in a 4th zone. So right now I can't even spin up a node in the last AZ since in my infrastructure, I'm not supporting it.
I think it may be more than just where there are or not nodes vs masters in an AZ. As an additional use case, consider pods that must run on a "dedicated" tenancy node in AWS, (which have a baseline cost per-zone) for compliance. Since masters are supposed to be an odd number (i.e. 3 zones) it would make sense to have 3 instead of 2. But for a pod that has to be scheduled on a "dedicated" tenancy node, there may be only 2 zones with nodes that are "dedicated" due to said cost. Even if there are nodes in all zones, a pod may need to be scheduled on a specific node or subset thereof. More information about node selection: More information about why this specific case would be necessary here: If PVs are used by pods, and pods must be able to access them from whatever nodes they are constrained/scheduled to, I think whatever solution is implemented needs to account for that.
do you have any other non-Kubernetes instances in us-east-1b in your AWS account? Did you try to tag your AWS instances that run Kubernetes with tag - all AWS instances in one Kubernetes cluster must share the same tag value, so you can run multiple Kubernetes clusters in one AWS account. This tag is not documented, see .
There is another non-k8s instance running in us-east-1b, but it's not related (created via breanstalk). I labels to all my nodes / controller as you sugessted and still it created the PV in . I may bail on Statefulset for now and manage my pv's by hand since I'm building out an operator and it wont be that painful, but would like to see if this could be addressed. I'm also willing to help out if you can help point into the right places this logic lives. I'll also try and dig in and see if I can find it myself.
It would be ideal to have the StorageClass accept an expression or wildcard zone under the aws provisioner. Ideally I shouldn't have to care where my pods are running and where the EBS are created.
We've a PR that adds "zones:" with list of zones instead of single "zone:". No wildcards though.
I just hit this as well. I think moving the "zone" decision closer to the provisioning decision would be best. Perhaps wait until a PVC is consumed by a pod before you create it in a zone. That way corner cases like this are not a problem.
Just want to check whether PR is good enough to solve the issue raised here?
, is right. Another use case is that if pod consumes 1 PVC, associated PVs might be created in different zones, therefore pod becomes unschedulable, as there is no node which belongs 1 AZ. Zone decision when provisioning PV must be made as late as possible, while currently it seems to be a very first thing to do.
I don't think it can be closed yet, reasons are: those magical tags are not documented anywhere, how people or going to know to use them? can't it continue to create volume in a zone where pod with a given PVC can never be scheduled (lets say it has node affinity configured to a node in a particular AZ)?
Would it make sense to say that the dynamic creation of a volume should happen only once a PVC has been attached to a pod which has been successfully scheduled to a node? Then the av_zone of the volume should match that of the selected node?
Yes I would agree with that statement. The PVC should be created when we know the node and AZ it is a apart of.
Just to add another use case to this - I just spun up a new cluster in AWS. Masters spanning , , and . After creating the ASG for nodes, I learned that AWS is out of instance capacity in for that particular instance type. Now I currently don't have nodes in (until capacity is restored and I can move an instance over). The reason this is different is because, in this case, I don't want to tell k8s to only use a subset of AZs. I want it to deploy across all AZs that are available. It just so happens that at this moment in time not all AZs are available. suggestion above would work, I believe. Schedule the node first, then provision a volume. That might cause other problems, though.
I just ran into this today - PVC created in eu-west-1c, but the only node with memory to to run the pod was in eu-west-1b. This was a new pod and a new volume claim. Can we have this re-opened at least please?
We had to ultimately get rid of all persistence volume claims on AWS because of this. For really large clusters with lots of reserved capacity in all availability zones that woulnd't be such a big deal, but I'd image most companies are trying to optimize their cost and don't have that many node pre-provisioned just for the sake of it. Still looking for a solution which is easy to configure and AZ-failure friendly
Right now, there isn't really. You have a few options: EFS - it is regional, not zonal. You have to be OK working with NFS as the protocol, and potential performance implications. To be fair, NFS has gotten quote good, I know companies using it to store critical data. I am not an NFS expert, so don't ask my opinions here... Commercial solutions - Portworx and StorageOS both provide local storage, backed by local disk or EBS, but replicated at the app level (their app) between nodes, across zones, so you don't depend on zonal resources (like EBS). DIY - have fun. Ceph, some other solutions. For one use case with which I am involved, we are living with it, and using EFS where necessary. We don't love it, but the resources to implement something else aren't there now.
Would in a StorageClass go some way to fixing the issue? (At least for new claims anyway)? From Edit: I tried this, and the kube-controller-manager logs say: but the pod is FailedScheduling "2 node(s) didn't find available persistent volumes to bind.".
I solved this in the Elastic Operator by creating StorageClasses per zone and spreading work across that way. You need to make sure your app is zone tolerant. For example, the Elastic operator sets up sharding and replication across zones, given one AZ goes down, the other two still have enough of the data to continue functioning.
doesn't each pod then need access to storage from a different class (appropriate for the zone it is in)? How does that work?
Just ran into this as well, running work load specific zones in 2 regions for development to save cost over 3, had an EBS volume provisioned in the zone with no instance.
should address the above scenarios when you use set in the storageclass. The plumbing for went into EBS in 1.12 as a beta feature.
I have a similar issue: 3 masters + 3 nodes in 3 regions. I have a PV (created on regionA) for data backups created by a CronJob. When the cron's job gets scheduled on a node on a different region (regionA's CPU has been allocated almost fully and the master chooses another node in a different region) - it can't start because the PV is not accessible. Is there a way to avoid this issue?
I am also facing the same issue with 1.10 version of Kubernetes. Nodes are running in two regions, PV got created in the same region. still, I am getting
As mentioned above, this feature is now in beta in 1.12:
Are there any workaround for Amazon EKS to address the above issue? I am trying to make a multi-az JupyterHub deployment and changing zones causes the "no available volume zone" error...
Is this still in progress? I just ran in to issue as well
we moved it to GCP with 1.10 version and didn't face the issue
I'm also having this issue... So there's no solution to this?
im also having this issue on AWS EKS Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 2m (x841 over 1h) default-scheduler 0/2 nodes are available: 1 Insufficient pods, 1 node(s) had no available volume zone. losing hope over here :(
Warning FailedScheduling 3m2s (x660 over 58m) default-scheduler 0/2 nodes are available: 1 Insufficient pods, 1 node(s) had no available volume zone.
Another similar case here, like 0/2 nodes are available: 1 Insufficient pods, 1 node(s) had no available volume zone. Like the use case by but the reason is not memory is pods allocation. The EBS was created in the AZ where the node is full (1c).. you can guess the rest..
Any update on this ? Since Node affinity is a beta feature causing this issue? Edit: (Beware: Updating your EKS cluster can potentially screw things up & cause nodes and volumes in different zones)
I'm using EKS to deploy my services integrating it with rancher, I ran into the same issue as mentioned above. My setup is something like this: have few labels for the worked nodes to seperate the nodes per namespace/team as each team requires different types of instances. created the nodes for the pods in that node needs persistant volumes in a single availability zone as aws EBS attachment is per availability zone. I deployed my applications, the persistant volume got created but in different availability zone as I used the default storage class which creates EBS in random zones. The issue is fixed by creating another storage class with only single availbility zone.
I am using EKS , and I designed not only my pods to be Cattle, but also my nodes are Cattle as well : I have a Pod created by statefulset. This statefulset defines a PVC. The PVC triggers the creation of new EBS in the same AZ of the Node (i don't have pb here) Because Nodes are Cattle (destoryable any time), it happen that there is no Node exist in the AZ of EBS (PVC) =Pod of statefulset remains in "Pending" state =Pod events shows this error: " 1 node(s) had no available volume zone" Once a New node joins the cluster in the wanted AZ, the Pod of statefulset runs like a charm because the EBS can be attached at that node (same AZ). Thinking to use EFS ( )but looking for better solutions.
from my experience, EFS performance is not that great. better avoid it.
It depends how you use it. It's not great for lots of little files (don't check out a git repo on it, for instance ). It's been working great for us in production for almost a year so far, mainly on larger PDF files etc. But, like anything, definitely test it for your use case.
So I do have a question. If I just put the PVC into the specific availability. zone. Does that allow the other pods on different nodes to use it?
The nodes running on other availability zones couldn't access this PV until if you use EFS rather than EBS
I ran into a similar issue today investigating a error I have a PVC: and originally it bound fine, with a PV in us-east-1d Then, I ran an upgrade and none of the new nodes were created in us-east-1d And, now I have a disk in a zone without any nodes.
I found the issue. The issue is that when you are using AWS then the PVC is set to the Node. That means that if you try to put that pod or deployment on another node then it will not connect with the PVCs and being a pending state. It's an AWS thing. I resolve this by specifying taints and toleration to the nodes within that specific region.
thank you! can you post an example for more elaboration

Discussion:
createTagSteps        = 9 ) <ins> // awsTagNameMasterRoles is a set of well-known AWS tag names that indicate the instance is a master // The major consequence is that it is then not considered for AWS zone discovery for dynamic volume creation. var awsTagNameMasterRoles = sets.NewString("kubernetes.io/role/master", "k8s.io/role/master") </ins> // Maps from backend protocol to ELB protocol var backendProtocolMapping = map[string]string{ "https": "https",

return matchingInstances, nil } <del> // getAllZones retrieves  a list of all the zones in which nodes are running </del> <ins> // getCandidateZonesForDynamicVolume retrieves  a list of all the zones in which nodes are running </ins> // It currently involves querying all instances <del> func (c *Cloud) getAllZones() (sets.String, error) { </del> <ins> func (c *Cloud) getCandidateZonesForDynamicVolume() (sets.String, error) { </ins> // We don't currently cache this; it is currently used only in volume // creation which is expected to be a comparatively rare occurrence.

zones := sets.NewString() for _, instance := range instances { <del> // Only return fully-ready instances when listing instances // (vs a query by name, where we will return it if we find it) if orEmpty(instance.State.Name) == "pending" { glog.V(2).Infof("Skipping EC2 instance (pending): %s", *instance.InstanceId) </del> <ins> // We skip over master nodes, if the installation tool labels them with one of the well-known master labels // This avoids creating a volume in a zone where only the master is running - e.g. #34583 // This is a short-term workaround until the scheduler takes care of zone selection master := false for _, tag := range instance.Tags { tagKey := aws.StringValue(tag.Key) if awsTagNameMasterRoles.Has(tagKey) { master = true } } if master { glog.V(4).Infof("Ignoring master instance %q in zone discovery", aws.StringValue(instance.InstanceId)) </ins> continue }

// CreateDisk implements Volumes.CreateDisk func (c *Cloud) CreateDisk(volumeOptions *VolumeOptions) (KubernetesVolumeID, error) { <del> allZones, err := c.getAllZones() </del> <ins> allZones, err := c.getCandidateZonesForDynamicVolume() </ins> if err != nil { return "", fmt.Errorf("error querying for all zones: %v", err) }