jiebi/CodeConvo huggingface dataset:
Code change:
framework.ExpectNoError(err) } <ins> func (l *ltrMgr) expandLocalVolumeBlockFS(ltr *LocalTestResource, mbToAdd int) error { ddCmd := fmt.Sprintf("dd if=/dev/zero of=%s/file conv=notrunc oflag=append bs=1M count=%d", ltr.loopDir, mbToAdd) loopDev := l.findLoopDevice(ltr.loopDir, ltr.Node) losetupCmd := fmt.Sprintf("losetup -c %s", loopDev) return l.hostExec.IssueCommand(fmt.Sprintf("%s && %s", ddCmd, losetupCmd), ltr.Node) } func (l *ltrMgr) ExpandBlockDevice(ltr *LocalTestResource, mbtoAdd int) error { switch ltr.VolumeType { case LocalVolumeBlockFS: return l.expandLocalVolumeBlockFS(ltr, mbtoAdd) } return fmt.Errorf("Failed to expand local test resource, unsupported volume type: %s", ltr.VolumeType) } </ins> func (l *ltrMgr) Create(node *v1.Node, volumeType LocalVolumeType, parameters map[string]string) *LocalTestResource { var ltr *LocalTestResource switch volumeType {

Related discussion:
text: There are use cases where users want to expand the local volume (may be managed by LVM or something similar), but even though they can edit underlying PV (and expand the underlying device) - the reported size in pvc.Status keeps reporting older value because in-tree local volumes are non-expandable. We could make this easier for users by - implementing interface for local volumes. This would allow following steps for local volume expansion: /admin edits the pvc - requesting new size. expands the underlying device. edits the PV and sets new size. will expand the filesystem and update This would at least get the user from impossible territory to possible. cc /sig storage
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Hi & any news on this issue ? The related PR seems stalled.
merged. So this is fixed now.
Oh right ! I didn't looked close enough and thought it was only for directory based volumes. Thank you.
Code change:
source "${KUBE_ROOT}/hack/lib/init.sh" source "${KUBE_ROOT}/hack/lib/util.sh" # This sets up the environment, like GOCACHE, which keeps the worktree cleaner. kube::golang::setup_env kube::util::ensure-temp-dir # Creating a new repository tree # Deleting vendor directory to make go-licenses fetch license URLs from go-packages source repository git worktree add -f "${KUBE_TEMP}"/tmp_test_licenses/kubernetes HEAD >/dev/null 2>&1 || true cd "${KUBE_TEMP}"/tmp_test_licenses/kubernetes && rm -rf vendor # Ensure that we find the binaries we build before anything else. export GOBIN="${KUBE_OUTPUT_BINPATH}" PATH="${GOBIN}:${PATH}" # Explicitly opt into go modules, even though we're inside a GOPATH directory export GO111MODULE=on <ins> function http_code() { curl -I -s -o /dev/null -w "%{http_code}" "$1" } </ins> allowed_licenses=() packages_flagged=()

Related discussion:
text: When I run it manually, I get: Clearly that is wrong, it's just looking in the wrong place. I also see a reference to which is still open 2 years later. Does this script run? Is it useful?
/sig testing
This job runs the script and fails as well:
If it ran fast, would we run it on every PR?
Another note: It runs around 5-6 minutes right now:
It has a very poorly placed n^2 loop. That can be fixed. Unfortunately the go-licenses tool itself is really slow, too.
It looks like that go-licenses cannot handle multi module repos if the license is only available in the repo root: The question is: do we want to contribute such a feature or don't we need the license test at all?
We should vet licenses in presubmit. Though I think dep approvers are pretty good at checking this, one less thing to manually vet would be ideal. If we can't speed it up, we can add a job that only runs on vendor/ changes.
/triage accepted
Code change:
func TestEmptyCPURequest(t *testing.T) { tc := testCase{ <del> minReplicas:     1, maxReplicas:     5, initialReplicas: 1, desiredReplicas: 1, CPUTarget:       100, reportedLevels:  []uint64{200}, useMetricsAPI:   true, </del> <ins> minReplicas:         1, maxReplicas:         5, initialReplicas:     1, desiredReplicas:     1, CPUTarget:           100, reportedLevels:      []uint64{200}, reportedCPURequests: []resource.Quantity{}, useMetricsAPI:       true, </ins> expectedConditions: []autoscalingv1.HorizontalPodAutoscalerCondition{ {Type: autoscalingv1.AbleToScale, Status: v1.ConditionTrue, Reason: "SucceededGetScale"}, {Type: autoscalingv1.ScalingActive, Status: v1.ConditionFalse, Reason: "FailedGetResourceMetric"},

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: We are starting to use the HPA (horizontal pod autoscaling) for one of our components. What is special about this particular component is that the warm up period is very long (sometimes even more than 30 minutes). During that time, the pod is in Running state but its containers are in Not Ready state. This pod is of course not able to receive any traffic so the metric we use in the HPA is ignoring those pods that are not Ready. The equation used by HPA to calculate the new number of replicas is as follows Our problem is that the equation used by HPA to calculate the new number of replicas is including in the "current replica count" the pods that are "Not Ready" so the value for "new replica count" is not correct. Since this warm up is so long we end up by having a lot of more instances of our services of what it should be necessary. They eventually get deleted once gradually the metrics start going down but that will take a lot of time and a big waste of resources without mentioning that with a changing traffic this HPA will never stabilize What you expected to happen: I would like to have a flag somewhere where I tell HPA to don't include in the "current replica count" the pods that are in "Not Ready" state How to reproduce it (as minimally and precisely as possible): Have a component that takes a lot of time to start and being able to receive traffic. Then define a metric that will ignore the pods that are Not Ready. Finally create a metric that will scale that component based on the mentioned metric Anything else we need to know?: Environment: Kubernetes version 1.7 Cloud provider or hardware configuration: Azure
/sig autoscaling
Hi - thanks for reaching out about this! It looks like this section of the , handles when pods aren't ready. I think you this behavior in , so I'm interested in your thoughts on whats best here. If we do end up adding a flag for this, I'd be happy to write the code :) Also, purely for my own curiosity, I'm curious what type of pod is taking 30 minutes to become ready. If your traffic pattern varies at a rate faster than 30 minutes, it may be difficult for auto-scaling to be helpful.
can you also share a bit more about your HPA?
Code change:
func TestEmptyCPURequest(t *testing.T) { tc := testCase{ <del> minReplicas:     1, maxReplicas:     5, initialReplicas: 1, desiredReplicas: 1, CPUTarget:       100, reportedLevels:  []uint64{200}, useMetricsAPI:   true, </del> <ins> minReplicas:         1, maxReplicas:         5, initialReplicas:     1, desiredReplicas:     1, CPUTarget:           100, reportedLevels:      []uint64{200}, reportedCPURequests: []resource.Quantity{}, useMetricsAPI:       true, </ins> expectedConditions: []autoscalingv1.HorizontalPodAutoscalerCondition{ {Type: autoscalingv1.AbleToScale, Status: v1.ConditionTrue, Reason: "SucceededGetScale"}, {Type: autoscalingv1.ScalingActive, Status: v1.ConditionFalse, Reason: "FailedGetResourceMetric"},

Related discussion:
text: What kind of metric type is it? Based on the description I'm suspecting that it is an Object metric, am I right?
So answer about why our process takes that much time to start. Basically this is a server that in order to do what he needs to do, he needs to load a lot of data in memory (several GBs) and then perform some warm up that are pretty intensive. The idea is that he doesn't need to grab (almost) anything from outside to attend a request so it's latency is extremely low (in the order of millisecond) By the way. Because of this we have another problem which is that there is not an specific flag per HPA object that we could tell k8s to don't try to scale up again passed some time so the new pods have enough time to get ready before the next scale happens. I know that there is a cluster wide flag but we will need to have a different one by resource that we scale. Now to answer You pointed another interesting topic. The metric we use is % Utilization. We are able to calculate the level of utilization of our pods and we want to keep this metric in an specific value. About the flag. I suggested the flag so it doesn't change the existing behavior by default but I honestly think that the default behavior should be to don't include the NonReady pods as part of the calculation
do you think we can add this change to 1.10? it's critical bug/feature for us and we are happy to help with code/testing.
It's really late for 1.10, especially since we don't have any code here, and I'd like to know more about what's going on here, and especially see if it reproduces on Kubernetes 1.9 (we've had HPA controller bugs in the past). FWIW, currently unready pods are treated as consuming 0% of their utilization on scale up, and it was intended that this should prevent such issues. That is, the formula is actually: If the pods are marked as not ready, I find it strange that they would be affecting the scaling that much, unless your other pods are under real heavy load (i.e. I would assume that the unready pods would skew the scaling back down so that the number of pods just stayed constant).
I believe there is different code branch here.
Code change:
func TestEmptyCPURequest(t *testing.T) { tc := testCase{ <del> minReplicas:     1, maxReplicas:     5, initialReplicas: 1, desiredReplicas: 1, CPUTarget:       100, reportedLevels:  []uint64{200}, useMetricsAPI:   true, </del> <ins> minReplicas:         1, maxReplicas:         5, initialReplicas:     1, desiredReplicas:     1, CPUTarget:           100, reportedLevels:      []uint64{200}, reportedCPURequests: []resource.Quantity{}, useMetricsAPI:       true, </ins> expectedConditions: []autoscalingv1.HorizontalPodAutoscalerCondition{ {Type: autoscalingv1.AbleToScale, Status: v1.ConditionTrue, Reason: "SucceededGetScale"}, {Type: autoscalingv1.ScalingActive, Status: v1.ConditionFalse, Reason: "FailedGetResourceMetric"},

Related discussion:
text: metric definition { "type": "Object", "object": { "target": { "kind": "Service", "name": "prometheus-exporter" }, "metricName": "PercentUtilization", "targetValue": "20" } } which should handle as and doesn't check Pod state.
I think proper handling of unready pods is a difficult problem that will require significant work. I suspect we may find out we need to start distinguishing pods that are booting up from pods that are 'broken' and it took us quite a while to solve analogous problem with nodes in Cluster Autoscaler (to be clear: I don't know if we will have to do that, that's just my intuition for now). That being said I agree the formula in GetObjectMetricReplicas will generally lead to scaling-up too much if the pod takes longer to initialize then backoff period in HPA (3 minutes). Assume it takes pod 4 minutes to boot up and we're currently running 2 pods. Metric value is 20 and target is 10. In this case: scales-up to 4 pods (20/10 utilization 2 pods). are starting up, they're not doing anything yet. 3 minutes later HPA scales-up to 8 pods (20/10 utilization 4 pods). If we only counted ready pods the scale-up in 3 would not occur, because the desired replicas would still be 4 (20/10 * 2). My intuition would be that this is the expected behavior of HPA. Also if there is no space in the cluster we will just keep scaling-up until we hit max replicas. The only case where current behavior would be more correct than filtering out unready pods in this formula is if someone uses a metric that is calculated based on total number of replicas. Do we advertise that as the right way to use HPA? If we add to Object metrics in 1.11 this usage pattern will stop making sense. To sum up - I think we will need to change the formula to only count ready pods at some point. I can see how current behavior could be a problem in any use-case except the one described in paragraph above. I'm inclined to agree with and provide some way to change the behavior in 1.10 (a flag or an annotation?), treating it as a bug fix.
Code change:
func TestEmptyCPURequest(t *testing.T) { tc := testCase{ <del> minReplicas:     1, maxReplicas:     5, initialReplicas: 1, desiredReplicas: 1, CPUTarget:       100, reportedLevels:  []uint64{200}, useMetricsAPI:   true, </del> <ins> minReplicas:         1, maxReplicas:         5, initialReplicas:     1, desiredReplicas:     1, CPUTarget:           100, reportedLevels:      []uint64{200}, reportedCPURequests: []resource.Quantity{}, useMetricsAPI:       true, </ins> expectedConditions: []autoscalingv1.HorizontalPodAutoscalerCondition{ {Type: autoscalingv1.AbleToScale, Status: v1.ConditionTrue, Reason: "SucceededGetScale"}, {Type: autoscalingv1.ScalingActive, Status: v1.ConditionFalse, Reason: "FailedGetResourceMetric"},

Related discussion:
text: WDYT
Ah, yeah, I missed that it was object metrics before. I'm not entirely convinced we need a flag here -- it might just make sense to bring Object metrics in line with all the other metrics types. On the other hand, there's a subtlety here -- the other metric types don't quite ignore unready pods -- they just ignore the metrics from unready pods, which causes a heavy bias in the direction of not scaling, but an overwhelming spike can still cause scaling. That behavior is not as achievable using object metrics. I'm inclined to just ignore that subtlety, though, and say that object metrics should ignore unready pods entirely.
I think this is what we should do long term. I was worried about breaking someone who relies on targetValue + averaged metric to get effect similar to targetAverageValue and though about providing something for transitional period, but if you're ok with it I'd rather just change it as you say.
I think it's safe to change as-is -- that feature's still in beta, and it's not an API change, so it shouldn't just hard-break anyone. Plus, we've change stuff like this before (when we did the initial resource metrics change).
As said, we are more than ready to help with the fix/test if required. This since this is a major blocking for us in the implementation of the HPA in our side
is anyone actively working on implementing this? If not, I'm happy to. I think I have a fairly clear picture of the change we need to make and will likely have time early this week to get it done.
(Also, from the conversation above, to we want to tag this as a release blocker for 1.10)?
I probably won't be able to work on this before the end of this week. Happy to do a review if you jump in and implement it.
I don't think this qualifies as a release blocking bug (but I'm willing to have someone try and convince me otherwise ;-)). There doesn't seem to be any behavior here that's explicitly broken -- it's just sub-optimal in some use cases. I'd definitely like to address this in 1.11, though.
it's not release blocker, but to share our experience ;).
Code change:
func TestEmptyCPURequest(t *testing.T) { tc := testCase{ <del> minReplicas:     1, maxReplicas:     5, initialReplicas: 1, desiredReplicas: 1, CPUTarget:       100, reportedLevels:  []uint64{200}, useMetricsAPI:   true, </del> <ins> minReplicas:         1, maxReplicas:         5, initialReplicas:     1, desiredReplicas:     1, CPUTarget:           100, reportedLevels:      []uint64{200}, reportedCPURequests: []resource.Quantity{}, useMetricsAPI:       true, </ins> expectedConditions: []autoscalingv1.HorizontalPodAutoscalerCondition{ {Type: autoscalingv1.AbleToScale, Status: v1.ConditionTrue, Reason: "SucceededGetScale"}, {Type: autoscalingv1.ScalingActive, Status: v1.ConditionFalse, Reason: "FailedGetResourceMetric"},

Related discussion:
text: without this feature we cannot use HPA as is and we need to figure out how to build/run fork (which can be challenging because everything inside controller-manager). also we are planning to upgrade to 1.10 quickly, but not sure for 1.11 (e.g. we are two versions behind now).
Sounds like a plan :) Will try and get this done tomorrow morning!
- I submitted a pr for this - would love your eyes on it so you can see if you think it meets your use case :)
/assign
GitHub didn't allow me to assign the following users: mattjmcnaughton. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I checked the code and it seems good to me. About the comment about scaling down if the readyPods<CurrentPods, we should be fine since our metric is already not including the pods that are not ready.
Is there any advance in this ticket? I'm wondering if this made it to k8s 1.10 or if we need to wait for k8s 1.11?
The PR is lgtm-ed and it will merge once the code freeze is lifted. Meaning it will go to 1.11. To get it into 1.10 you would need to get approval for milestone from (sig-autoscaling lead).
Code change:
// since volumes should be removed from this list as soon a detach operation // is considered, before the detach operation is triggered). GetVolumesToReportAttached() map[types.NodeName][]v1.AttachedVolume <ins> // GetNodesToUpdateStatusFor returns the map of nodeNames to nodeToUpdateStatusFor GetNodesToUpdateStatusFor() map[types.NodeName]nodeToUpdateStatusFor </ins> } // AttachedVolume represents a volume that is attached to a node.

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: $ kubectl describe pods test-ipj93 Name: test-ipj93 Namespace: default Node: kubernetes-minion-group-yxra/10.128.0.3 Start Time: Mon, 28 Nov 2016 14:39:28 -0800 Labels: <noneStatus: Pending IP: Controllers: <noneContainers: busybox: Container ID: Image: busybox Image ID: Port: Command: sh -c while true; do echo (hostname)-(cat /proc/stat cut -f 4 -d" "); sleep 30; done QoS Tier: cpu: Burstable memory: BestEffort Requests: cpu: 100m State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment Variables: Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: test ReadOnly: false default-token-kjoqg: Type: Secret (a volume populated by a Secret) SecretName: default-token-kjoqg Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 57m 57m 1 {default-scheduler } Normal Scheduled Successfully assigned test-ipj93 to kubernetes-minion-group-yxra 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulling pulling image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulled Successfully pulled image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Created Created container with docker id ; Security:[seccomp=unconfined] 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Started Started container with docker id 47m 47m 6 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.
Code change:
// since volumes should be removed from this list as soon a detach operation // is considered, before the detach operation is triggered). GetVolumesToReportAttached() map[types.NodeName][]v1.AttachedVolume <ins> // GetNodesToUpdateStatusFor returns the map of nodeNames to nodeToUpdateStatusFor GetNodesToUpdateStatusFor() map[types.NodeName]nodeToUpdateStatusFor </ins> } // AttachedVolume represents a volume that is attached to a node.

Related discussion:
text: SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp: lookup kubernetes-master on 169.254.169.254:53: no such host 46m 46m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: i/o timeout 45m 45m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: getsockopt: connection refused 11m 11m 2 {controller-manager } Warning FailedMount Failed to attach volume "test" on node "kubernetes-minion-group-yxra" with: error getting instance "kubernetes-minion-group-yxra" 9m 9m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 9m 9m 2 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig.
Code change:
// since volumes should be removed from this list as soon a detach operation // is considered, before the detach operation is triggered). GetVolumesToReportAttached() map[types.NodeName][]v1.AttachedVolume <ins> // GetNodesToUpdateStatusFor returns the map of nodeNames to nodeToUpdateStatusFor GetNodesToUpdateStatusFor() map[types.NodeName]nodeToUpdateStatusFor </ins> } // AttachedVolume represents a volume that is attached to a node.

Related discussion:
text: This is most likely due to lack of PodCIDR] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedMount Unable to mount volumes for pod "test-ipj93_default(-b5bb-11e6-bab0-)": timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] What you expected to happen: GCE PD to attach and mount successfully post the upgrade How to reproduce it (as minimally and precisely as possible): Running the upgrade steps in the doc for upgrade of manual test from 1.4 to 1.5 Anything else do we need to know:
Ran the upgrade again. This time the .
kube-controller-
Current theory is because kubelet deletes and recreates the node API object during upgrade (because externalID changed), see the new object does not contain the correct status (it is written by the Attach/Detach controller to the old object but not subseqently updated). If this proves to be the issue, the proposed fix is to force the attach/detach controller to update the node status when the node's external ID changes. is working to verify this theory by executing this code path (without a full upgrade). Once he has validated the theory, he will work on the fix, verify it, merge it to master, and get it cherry-picked to 1.5.
: is expected to fix this problem?
: Yes