Change:
E0513 22:55:45. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 266 E0513 22:55:45. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 266 E0513 22:55:48. 957 ] watch of api.Secret ended with: Stop requested E0513 22:55:48. 957 ] watch of api.ServiceAccount ended with: Stop requested E0513 22:55:48. 957 ] watch of api.Namespace ended with: Stop requested E0513 22:55:48. 957 ] watch of api.ServiceAccount ended with: Stop requested E0513 22:55:48. 957 ] watch of api.Secret ended with: Stop requested E0513 22:55:48. 957 ] watch of api.ServiceAccount ended with: Stop requested W0513 22:55:49. 957 ] Resetting endpoints for master service "kubernetes" to &{{ } {kubernetes default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 6443 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes" to &{{ } {kubernetes default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 6443 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} E0513 22:55:50. 957 ] unable to sync kubernetes service: endpoints "kubernetes" cannot be updated: the resource was updated to 279 E0513 22:55:50. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 280 E0513 22:55:50. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 280 E0513 22:55:50. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 280 E0513 22:55:50. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 280 W0513 22:55:51. 957 ] Portal net unspecified. Defaulting to 10.0.0.0/24. I0513 22:55:51. 957 ] Will report 10.0.0.22 as public IP address. E0513 22:56:00. 957 ] Unable to authenticate the request due to an error: Token has been invalidated E0513 22:56:00. 957 ] Unable to authenticate the request due to an error: Token has been invalidated E0513 22:56:00. 957 ] Unable to authenticate the request due to an error: Token has been invalidated E0513 22:56:01. 957 ] Unable to authenticate the request due to an error: Token has been invalidated E0513 22:56:08. 957 ] watch of api.Secret ended with: Stop requested E0513 22:56:08. 957 ] watch of api.ServiceAccount ended with: Stop requested E0513 22:56:08. 957 ] watch of api.Namespace ended with: Stop requested E0513 22:56:08. 957 ] watch of api.ServiceAccount ended with: Stop requested E0513 22:56:08. 957 ] watch of api.Secret ended with: Stop requested E0513 22:56:08. 957 ] watch of api.ServiceAccount ended with: Stop requested FAIL FAIL 111.315s
Do we know which test is failing? The last messages in the log are from , but I don't actually see a test failure message printed, and I can't recreate any failures. The "Unable to authenticate the request due to an error: Token has been invalidated" messages are expected since the test exercises attempting to use invalid tokens () Also, the "Stop requested" messages indicate that particular test completed and invoked the deferred stopFunc, which shuts down the controllers and closes the test server
Here is the failing test: --- FAIL: TestPersistentVolumeClaimBinder (3.45s) Expected claim to be bound: &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:claim02 GenerateName: Namespace:default SelfLink:/api/v1beta1/persistentVolumeClaims/claim02?namespace=default UID:-f9c3-11e4-96ec- ResourceVersion:172 CreationTimestamp:2015-05-13 22:54:47 +0000 UTC DeletionTimestamp:<nilLabels:map[] Annotations:map[]} Spec:{AccessModes:[ReadOnlyMany ReadWriteOnce ReadWriteMany] Resources:{Limits:map[] Requests:map[storage:{Amount: Format:BinarySI}]} VolumeName:} Status:{Phase:Pending AccessModes:[] Capacity:map[]}}
you aware of the issue with TestPersistentVolumeClaimBinder?
I ran into above issue a lot.
Sorry, hadn't seen this sooner. Lost in the flurry of email traffic. I am looking into it now.
I cannot reproduce on F20, Go 1.3.1 I'll try 1.4 next. Any other info would be appreciated.
Here is the output of docker version from my desktop. Client version: 1.6.2 Client API version: 1.18 Go version (client): go1.4.2 Git commit (client): OS/Arch (client): linux/amd64 Server version: 1.6.2 Server API version: 1.18 Go version (server): go1.4.2 Git commit (server): OS/Arch (server): linux/amd64 I started to run into this issue two days ago, and just ran into it half hour ago right after I sync-ed to HEAD again. Any more information you need here?
Thanks :)
I cannot reproduce this on F20, Go 1.4.2 Any runtime settings I might be missing? I see hack/test- is set to "-race" .

Discussion:
t.Errorf("Expected match but received nil volume for scenario: %s", name) } if scenario.expectedMatch != "" && volume != nil && string(volume.UID) != scenario.expectedMatch { <del> t.Errorf("Expected %s but got volume %s instead", scenario.expectedMatch, volume.UID) </del> <ins> t.Errorf("Expected %s but got volume %s in scenario %s", scenario.expectedMatch, volume.UID, name) </ins> } if scenario.expectedMatch == "" && volume != nil { t.Errorf("Unexpected match for scenario: %s", name)

} } <ins> func TestMatchingWithBoundVolumes(t *testing.T) { volumeIndex := NewPersistentVolumeOrderedIndex() // two similar volumes, one is bound pv1 := &api.PersistentVolume{ ObjectMeta: api.ObjectMeta{ UID:  "gce-pd-1", Name: "gce001", }, Spec: api.PersistentVolumeSpec{ Capacity: api.ResourceList{ api.ResourceName(api.ResourceStorage): resource.MustParse("1G"), }, PersistentVolumeSource: api.PersistentVolumeSource{ GCEPersistentDisk: &api.GCEPersistentDiskVolumeSource{}, }, AccessModes: []api.PersistentVolumeAccessMode{api.ReadWriteOnce, api.ReadOnlyMany}, // this one we're pretending is already bound ClaimRef: &api.ObjectReference{UID: "abc123"}, }, } pv2 := &api.PersistentVolume{ ObjectMeta: api.ObjectMeta{ UID:  "gce-pd-2", Name: "gce002", }, Spec: api.PersistentVolumeSpec{ Capacity: api.ResourceList{ api.ResourceName(api.ResourceStorage): resource.MustParse("1G"), }, PersistentVolumeSource: api.PersistentVolumeSource{ GCEPersistentDisk: &api.GCEPersistentDiskVolumeSource{}, }, AccessModes: []api.PersistentVolumeAccessMode{api.ReadWriteOnce, api.ReadOnlyMany}, }, } volumeIndex.Add(pv1) volumeIndex.Add(pv2) claim := &api.PersistentVolumeClaim{ ObjectMeta: api.ObjectMeta{ Name:      "claim01", Namespace: "myns", }, Spec: api.PersistentVolumeClaimSpec{ AccessModes: []api.PersistentVolumeAccessMode{api.ReadOnlyMany, api.ReadWriteOnce}, Resources: api.ResourceRequirements{ Requests: api.ResourceList{ api.ResourceName(api.ResourceStorage): resource.MustParse("1G"), }, }, }, } volume, err := volumeIndex.FindBestMatchForClaim(claim) if err != nil { t.Fatalf("Unexpected error matching volume by claim: %v", err) } if volume == nil { t.Fatalf("Unexpected nil volume.  Expected %s", pv2.Name) } if pv2.Name != volume.Name { t.Errorf("Expected %s but got volume %s instead", pv2.Name, volume.Name) } } </ins> func TestSort(t *testing.T) { volList := NewPersistentVolumeOrderedIndex() for _, pv := range createTestVolumes() {

}, { ObjectMeta: api.ObjectMeta{ <ins> UID:  "gce-pd-20", Name: "gce004", }, Spec: api.PersistentVolumeSpec{ Capacity: api.ResourceList{ api.ResourceName(api.ResourceStorage): resource.MustParse("20G"), }, PersistentVolumeSource: api.PersistentVolumeSource{ GCEPersistentDisk: &api.GCEPersistentDiskVolumeSource{}, }, AccessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, api.ReadOnlyMany, }, // this one we're pretending is already bound ClaimRef: &api.ObjectReference{UID: "def456"}, }, }, { ObjectMeta: api.ObjectMeta{ </ins> UID:  "nfs-5", Name: "nfs002", },

return nil, err } <del> i := sort.Search(len(volumes), func(i int) bool { return matchPredicate(pv, volumes[i]) }) if i < len(volumes) { return volumes[i], nil </del> <ins> // volumes are sorted by size but some may be bound. // remove bound volumes for easy binary search by size unboundVolumes := []*api.PersistentVolume{} for _, v := range volumes { if v.Spec.ClaimRef == nil { unboundVolumes = append(unboundVolumes, v) } } i := sort.Search(len(unboundVolumes), func(i int) bool { return matchPredicate(pv, unboundVolumes[i]) }) if i < len(unboundVolumes) { return unboundVolumes[i], nil </ins> } return nil, nil }

}, }, } <del> return pvIndex.Find(pv, filterBoundVolumes) </del> <ins> return pvIndex.Find(pv, matchStorageCapacity) </ins> } // FindBestMatchForClaim is a convenience method that finds a volume by the claim's AccessModes and requests for Storage

// matchStorageCapacity is a matchPredicate used to sort and find volumes func matchStorageCapacity(pvA, pvB *api.PersistentVolume) bool { <del> // skip already claimed volumes if pvA.Spec.ClaimRef != nil { return false } </del> aQty := pvA.Spec.Capacity[api.ResourceStorage] bQty := pvB.Spec.Capacity[api.ResourceStorage] aSize := aQty.Value() bSize := bQty.Value() return aSize <= bSize } <del> // filterBoundVolumes is a matchPredicate that filters bound volumes before comparing storage capacity func filterBoundVolumes(compareThis, toThis *api.PersistentVolume) bool { if compareThis.Spec.ClaimRef != nil || toThis.Spec.ClaimRef != nil { return false } return matchStorageCapacity(compareThis, toThis) } </del>