Change:
// Images used for ConformanceContainer are not added into NodeImageWhiteList, because this test is // testing image pulling, these images don't need to be prepulled. The ImagePullPolicy // is v1.PullAlways, so it won't be blocked by framework image white list check. <del> imagePullTest := func(image string, hasSecret bool, expectedPhase v1.PodPhase, expectedPullStatus bool) { </del> <ins> imagePullTest := func(image string, hasSecret bool, expectedPhase v1.PodPhase, expectedPullStatus bool, windowsImage bool) { command := []string{"/bin/sh", "-c", "while true; do sleep 1; done"} if windowsImage { // -t: Ping the specified host until stopped. command = []string{"ping", "-t", "localhost"} } </ins> container := ConformanceContainer{ PodClient: f.PodClient(), Container: v1.Container{ Name:            "image-pull-test", Image:           image, <del> Command:         []string{"/bin/sh", "-c", "while true; do sleep 1; done"}, </del> <ins> Command:         command, </ins> ImagePullPolicy: v1.PullAlways, }, RestartPolicy: v1.RestartPolicyNever,

Discussion:
There is now a test tag from I remember a tag being approved in SIG Architecture and SIG Testing, but I don't remember a tag. If this was approved I think we need to broadcast it more widely as many existing test setups including current conformance testing appear to now need to skip / exclude this tag. cc /area conformance /sig architecture /sig testing
/assign
bug may not be the appropriate type, but seemed like the closest issue template to start from initially.
TL;DR nope, that was not something that I'm aware of.
cc
The email thread is in
the in that thread says: That's not the case here.
we should file a PR marked 1.14 and to yank that tag, just so we don't forget. Of course if the decision changes in the email thread, we can revisit the PR. ok?
makes sense to me
Catching up. Quick comment before I read the threads: Documented kinds of tests are currently here: WindowsOnly had been hypothesized, but not yet to that list.
Also, are there already tests tagged as SIG Windows?
On conformance tests: Windows isn't part of conformance yet, so no conformance test should have this tag. And no conformance test should contain any Skip directives. They should die in that case:
I looked at and don't understand why the Windows image test wasn't written as a separate test. To avoid either refactoring or copy/pasting? It looks like a pretty obscure approach -- auto-skipping individual test cases that contain a particular tag?
Pretty much. But it is no longer the case, since all those "should pull image from X" test cases have been refactored. (look at to see how it looks now)
Yes, that's better, thanks. To answer my own question regarding , here's an example: which also invokes inside the test. So we're agreed that we don't need yet?
now removes all instances of the [WindowsOnly] tag.

<!-- Please only use this template for submitting enhancement requests --What would you like to be : /sig testing /sig windows In the the PR we're adding a test case for pulling a Windows image from dockerhub (. However, there are no known public Windows images on , so the only option would be to upload it ourselves. Our proposal is to have it uploaded to the using Windows images). The proposed image manifest list to be uploaded is built from [1], which is currently hosted on dockerhub [2] and is currently used for testing Kubernetes with Windows nodes [3]. The manifest list contains the images: e2eteam/busybox:1.29-1803 (for Windows Server 1803) e2eteam/busybox:1.29-1809 (for Windows Server 2019) [1] [2] [3] Why is this needed: The Windows images will be used for the Windows equivalent test case for the following test:
/assign
/area conformance
It is my contention that we really only want one prod repo and that should be folded into the main repo. The authenticated repo should be for testing authenticated pulls and nothing else. To that end, I propose that we start with this. Does the image need to be named or tagged or something to indicate OS? How about * ? Is that too deep? Does it need Windows in the path or should we put it in the tag? Or can we have a multi-os manifest list? Also, is busybox the right image for testing, or can we do something even more minimal (to address the "what f there's a busybox CVE" question)?
Yes, (all images to date have used ), so I'd prefer that route over yet another name or tag. I'd also prefer something other than busybox if possible. I originally created because there were so many images in and it was hard to tell which of them were production images, which were test images, and which had been completely abandoned. Also, only a small number of Googlers could push to this repo since it contained all the production images. To address the first issue, I decided that all e2e test images would be sourced from the directory. To address the second issue, I created a new project with a wider ACL (though still currently limited to Googlers).

All that said, given the automation we're building for the main GCR repo, it'd probably be fine to fold these images back in.
The image promoter means that a single repo should be BETTER managed than individual repos - we will have a PR trail of who put images there and why.
We should probably broaden the discussion to non-busybox Windows test images too. (related: ). Do we have a lower bar for test images? If so, having a separate repo (or making it clear that the images are test only) does have clear benefits.
Do we want a lower bar? I don't think we do.
Nobody wants a lower bar until they want to add/update the image quickly for a new test, or fix a test failure to unblock the release. If we do effectively hold the same bar today, this shouldn't be a problem.
i agree. also we don't need schema v2. multi-os, multi-arch image here. just (e.g.) a nanoserver image that satisfies a Windows 1803 and Windows 1809 host. a nanoserver image will obviously imply Windows only. for a test called "should be able to pull image from [WindowsOnly]": all we need is to upload this image to [x] and i guess we need to decide on if we want this test to be / passing for Windows Containers GA.
And indeed, a manifest list with nanoserver images could be an alternative (considering that the ) (we did try to find a command which works on both OSes, and ultimately found curl, but not all Linux images contain curl, check PR ).
an official nanoserver image is a more secure download. do we need to build a nanoserver manifest list or are there official images that we can use? i would like to bring a topic here related to pause. why are all the pause images for Windows that i found 1.1 GB or more? we definitely need the absolute minimal pause container uploaded in a location where users can download it and use that location in the guide. AFAIK, 100 MB is as low as we can go for a native Windows container, but that's still much better... i brought this topic in the other PR. we don't need a command that works on both OSes.

if is the command to use on Windows containers only, so be it.
And also, my image? It might be out-of-date.
we need to start planning how Windows containers can used be easily on Windows nodes. the pause image should be pre-built, small and easy to pull by deployment tools, such as kops, kubeadm. that is much better, but still a bit too much? can we go down to ~100MB? this image size was reported by a user in a k/k issue recently: my assumption is that the user was building the image himself.
i am ok to have that conversation, but please file a ticket and let's discuss that in sig-windows
everyone, thank you for your help and responses. trying to bring back this conversation to the original request. what would you like us to do in order for the few folks from google that have permissions to upload the windows images to be used for testing.
I am happy to help publish this via the alias, which I believe is the correct long-term path. We need an image - preferably not one as prone to CVEs as busybox historically is. Hence a suggestion for "pause" or something equivalently useless and tiny (if it is tiny). Then, we need a path. First, we'll be setting precedent for test images, so I need input on that. I propose as the root. Objections from test folks? Second, do we want to squirrel this away as a Windwows-specific binary, or should we not name it in any particular way, so that we can pull the same image for all tests via a multi-os, multi-arch manifest list (which can be done later). E.g. vs How about * ? Is that too deep? Does it need Windows in the path or should we put it in the tag? Michael Michael <:
So, I'll sumarize the previous discussions. The proposed images are: The image currently named image tags:. based on: respectively. image is currently used in Windows E2E tests and should already exist on the nodes, so download should be minimal. The image. a manifest list needs to be created containing the tags: The image is used to tag the image, which is used by kubelet, so it should already exist on the nodes. smaller in size (~350MB).

the test should create pods with the command . I'm fine with either one, although I think people will tend towards the nanoserver image, and in fairness, I think it will be the easiest one to maintain long-term. AFAIK, Microsoft updates their container images periodically, so, we should do the same for the image. If not, the test using the image will then have to pull the entire image (350 MB is ok, IMO). E.g. vs Well, a manifest list can contain both Linux and Windows images, So, if the busybox image is picked, the Linux busybox image can be included in the manifest as well. Ideally, most images work on both OSes (with some exceptions, like the nanoserver or alpine images), so I don't think there's a need to put windows in the path. And if a new root path is being for new test images, I would like to ask that images mustn't container slashes in their names (e.g.:). Docker doesn't support images with slashes in their names (see ). Even if all the images on eventually support Windows, it would still be a good idea to be able to use dockerhub for testing purposes.
We need an image - preferably not one as prone to CVEs as busybox WHOAH. That's a 2GB image. For testing. I am in favor of this just based on the size (I was thinking something like kuberneets' pause, which is 240 kB. Over 1000 times smaller. We have different comprehensions of what "nano" means. 350 MB is not OK. It's acceptable if we have no other choice, but honestly we should create our own that is as small as humanly possible. Then, we need a path. First, we'll be setting precedent for test images, so I need input on that. I propose as the root. Objections from test folks? I don't have enough context on e2e future, so I'd like some input from someone who does. Second, do we want to squirrel this away as a Windwows-specific binary, or should we not name it in any particular way, so that we can pull the same image for all tests via a multi-os, multi-arch manifest list (which can be done later). E.g.

vs I agree (except for busybox, which is a known vector for contaigion). Having a multi-os, multi-arch "serve_hostname" or something equally dumb seems ideal. Docker HUB doesn't. Docker itself does, and GCR does. If you're using dockerhub for testing, you have to change "" to "e2eteam/whatever", right? So what does it matter if the GCR path is longer? That said, I would be OK to have a single dir if that is what test-team wants. It just seems likely to get messy eventually.
Yes, dockerhub, I forgot the "hub" :) And yes, that's what we're doing and it works for the most part. But, if you take a look here [1], basically, the root is , which, as you know, is not accepted by dockerhub. So, the proposal was to not have "image names" with slashes in them (e.g.: in this case). [1]
I see. I think it would be OK to ask that images don't use subdirs, but the e2eRegistry variable could. e.g. // Docker hub e2eRegistry = "e2eteam" // GCR e2eRegistry = "" ResourceController = Config{e2eRegistry, "resource-consumer-controller", "1.0"} That seems workable, though it doesn't answer the question of the actual path in GCR :) Claudiu Belu <:
That seems workable, though it doesn't answer the question of the Yep, exactly right. :)
what are the next steps on this ticket to get the images to
I am still waiting for which image we are going to use, and for any objections to "" as a root. Michael Michael <:
Well, I think we are in agreement that we should use the nanoserver images. As for the root repo, I have no objections.
can you please explicitly give all the images to upload in an authoritative list? i am hoping we can upload all of these by monday EOD.
Sure. As we've discussed previously, the images are:
no manifest list image that supports both 1809 and 1803? are we OK for only having 1809 for that "pull from GCR" test?

Yeah, I meant those images should be included in a single manifest list (something like or something), like we've previously discussed. As for the second question, AFAIK, the GCE Windows CI is using Windows Server 1803 at the moment, so it would be nice to have the new test running on that too.
see above for the two images we want to push. we also need to repush this image resource-consumer/controller:1.0 to avoid the "/" as resource-consumer-controller cc:
i pulled these images on Windows: size of each is roughly ~350MB. and then saved them to TAR and uploaded them here: to go lower than that we might have hack custom images that trim, for instance a bunch of DLLs that we don't need (for e.g. a minimal pause image), however such a custom image will just require yet another unique download if the users are already basing their workload containers on official images. also for such hacks we need someone to verify if the nanoserver LICENSE allows that. i'm pretty sure it doesn't.
no hacks please. we need to upload the images as they officially come from microsoft. :)
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
The slash thing is kinda irrelevant to me, it's just a limitation of Docker Hub as pointed out
i understand that. but can you make forward progress in testing that autoscale ticket without this work?
i think you have the two images to upload. let us know when this is complete and thank you so much!
The tars shouldn't be necessary. The nanoserver images include 2 layers that are type and should not be pushed. here's an example manifest from the bazel repo. you could probably use the existing rulesdocker to push nanoserver with no changes if you're building and pushing images with bazel
we can't follow this process due to Windows licensing. i will hide your post so that it is not followed by accident.
just to clarify - you can reference those in manifest lists, and docker push / pull them without actually pushing the layer contents to a manifest v2 complaint registry. That's the right way to do it.

That separates out user-owned files in the upper layers from those in the base layers that are always served from a Microsoft server due to licensing restrictions
Oh ok then. Keeping in mind the fact that the nanoserver image is a manifest itself (for amd64 and arm) and that you cannot reference another manifest list in your manifest list, then the the actual images which should be to the manifest list are: 1803 doesn't have multiarch.
the post is still quoted in your response. best to edit that as well? ok, looks like the licensing of these image is slightly more complicated than expected... i hope a tutorial is not required when pushing to GCR. :)
What is the status of this now? I am THOROUGHLY confused
me too according to here and here and as far as i understood, we only have to push the JSON files that hold layer references, but not the actual layers because this violates the licensing of the nanoserver images.
I'll post explicit steps using , etc.
a PowerShell window as an administrator Docker-CE 18.09.2 experimental mode for the Docker CLI Here's the copy & paste steps to pull the existing images, push them to the new registry, and create a manifest. Change $destinationRepo to be something like or your private repo name. The pushes will be nearly instantaneous because they're manifest-only pushes. All layer data is skipped. Now, you can inspect the manifest and it will show multiple images with different os.version And last, push it
Here's the example repo I made on When things are done this way, the original Windows image is still served from Microsoft as required by the license. If I were to add unique layers on top of this image - that content would be pushed to the registry.
Pushed to my staging, will try to get the promoter running ASAP
hmm, it looks like our image promoter is not quite ready for multi-platform yet. Let me try pushing to
yeah, you might be bumping into
Someone test that pull?
does that mean no Windows images can live on GCR? Am I misreading it?
I believe GCR supports it, but some of our tooling (which uses google/containerregistry) may not. OTOH I think tool is just using directly, so it might be unaffected.
!
Awesome, so that seems to work for now.

I guess the next step is to FROM that in your own containers.
or just use it in your tests. Do we close this for now, then?
SGTM, leaving the close call to
This looks right. I did a pull on a 1809 machine and the hashes match. is this enough to finish that test case, or are more images needed? We definitely need to get the images into a CI for 1.15. See /
this should be enough in terms of the test case.
^ FYI, there are now Windows test images at GCR. thanks !
Awesomeness, I've made a PR adding the testcase using this new image. This issue will close automatically once that merges. Test works for me. One question remains though. Do we have to have a Windows equivalent for the , so it fails on Windows. If we have to add a Windows equivalent for that too, then the nanoserver manifest list should also be pushed to the registry as well. Anyways. thanks for the help, everyone!
are you asking if can push the nanoserver manifest to the authenticated-image-pulling registry as well? Tim?
Yep. It is a conformance test, so it would be nice to be able to cover it as well.
any chance you push the nanoserver manifest to the authenticated-image-pulling registry as well? thanks for all your help! once you do the above, we can lose this issue
Try ""
thanks it works! !
please, close the issue once you check this.
thanks Tim! you rock!
can you please close this issue if we are done?
The issue is supposed to close when the PR merges. Closing anyways.