Code change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: On a fresh Ubuntu 16.04.3 system booted from the official cloud image, fails because exists. What you expected to happen: kubeadm successfully initializes the cluster How to reproduce it (as minimally and precisely as possible): a new VM from the latest the Anything else we need to know?: Contents of : Environment: Kubernetes version (use ): Cloud provider or hardware configuration: Hyper-V generation 1 virtual machine OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: none Others: none
This is related to the location where the kubelet persists its certificates while running in the background, waiting for config: the kubeadm 1.8.0-0 package accidentally used a location that is the kubeadm 1.8.0-1 package corrected that to use , but starting the kubelet before running causes files to be generated into the folder since expects there to be a running kubelet prior to being called, it shouldn't expect the kubelet's folder to be empty if you are scripting bootstrapping a known clean machine, there are a few possible workarounds until is released in 1.8.1 (any of the following work around this issue): verify this is the only preflight check failure, then run the init or join command with stop the kubelet service and remove prior to running the init or join command run prior to running init or join addressed as part of
cc
I did it worked for me!
On a known clean environment, that's an ok workaround for the moment, but the reason checks for existing files/folders is to avoid stomping an existing installation, and running would negate those checks.
(If you wonder why the deployment scripts that you wrote yesterday stopped working this morning – this is why! :-)) For the time being, before running or , I'll: check if or exists if they don't exist, stop kubelet, and wipe out I'd love to know if there are foreseeable pitfalls in this approach. Thanks! <3
apologies for the churn. we traded a for this false-positive fail safe issue. working with to determine the best way to resolve this.
No worries, and thanks for the super-quick turnaround time. Much appreciated! (I didn't mean to sound snarky or anything; sorry if that how it came out!)
This guys are posting new versions without properly testing it. Can you imagine brand new centos 7 installation and then getting: "kubeadm 1.8.0 init fails with "/var/lib/kubelet is not empty" how disgusting that was and what I was thinking about responsibility and professionalism here?
Everything was done by: but now it is even getting better: The HTTP call equal to 'curl -sSL http://localhost:/healthz' failed with error: Get http://localhost:/healthz: dial tcp [::1]:: getsockopt: connection refused you should remove 1.8 and put it do "back to the drawing board" stadium. It is useless even to test this 1.8
I apologize for the issue. Our top priority was to prevent data loss and issues upgrading existing installations that would disrupt already-running workloads. We are working now to resolve this issue with fresh installations.
Wish you luck.
ETA for a fix for this? should i hard code this deletion of contents of /var/lib/kubelet prior to running kubeadm init in my automation scripts? or simply add the ignore pre flight checks arguement will do the trick?
there are plenty of other pre-flight checks that are valuable and you don't want to skip, it's just the check for an empty that is incorrect. if you are scripting bootstrapping a known clean machine, there are a couple possible workarounds until is released in 1.8.1: run / skipping preflight checks stop the kubelet service, remove , then run /
ok just to confirm, moving forward, ideally should not be checking for contents in (i.e. having contents is totally fine), or should installing kubelet not populate contenst of ?
Correct. The instructions start the kubelet and let it run in a crash loop in the background, waiting for config. In that state, the kubelet is free to write to its directory containing state (), so should not require that directory to be empty in order to run
(Reposting my earlier tip since it worked like a charm for my automated deployment scripts.) For the time being, before running or , I'll: check if or exists if they don't exist, stop kubelet (), and wipe out () This might or might not be appropriate for your use-cases, but for mine it works great (I'm automatically deploying hundreds of k8s clusters of training purposes). Stopping kubelet is necessary to avoid race conditions where it would recreate the directory before you run .
If i wipe out the /var/lib/kubelet/pki directory and do not restart kubelet, the kubeadm init process hangs with kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:/healthz/syncloop' failed with error: Get http://localhost:/healthz/syncloop: dial tcp [::1]:: getsockopt: connection refused. [kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:/healthz' failed with error: Get http://localhost:/healthz: dial tcp [::1]:: getsockopt: connection refused. [kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:/healthz/syncloop' failed with error: Get http://localhost:/healthz/syncloop: dial tcp [::1]:: getsockopt: connection refused. [kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:/healthz' failed with error: Get http://localhost:/healthz: dial tcp [::1]:: getsockopt: connection refused. Unfortunately, an error has occurred: timed out waiting for the condition This error is likely caused by that: The kubelet is not running The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled) There is no internet connection; so the kubelet can't pull the following control plane images: You can troubleshoot this for example with the following commands if you're on a systemd-powered system: 'systemctl status kubelet' 'journalctl -xeu kubelet' When i check the status of kubelet, it's restarting and never starts because So, i am guessing my only option is to skip preflight checks, let me try that out..
Wish you luck. This is not the only problem: kubectl cluster-info Kubernetes master is running at http://localhost:8080 [root ~]# kubectl apply -f W1004 06:49:56. 2422 ] Failed to download OpenAPI (Get http://localhost:8080/swagger- dial tcp [::1]:8080: getsockopt: connection refused), falling back to swagger The connection to the server localhost:8080 was refused - did you specify the right host or port? Why did you put this 1.8 version when it is obviously not properly tested. Or tested at all?
Not to forget: systemctl status firewalld firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled) Active: inactive (dead) Docs: man:firewalld(1)
kubeadm does not run kubernetes unsecured. following the setup instructions in the output of configures to speak to the initialized server:
Ok. This worked. Thank you. Node also by running kubeadm join.... Now the problem is dashboard. I have followed: but still don't understand where to start kubectl proxy and max.I god when I try to access dashboard is (https://192.168.4.10:6443/): { "kind": "Status", "apiVersion": "v1", "metadata": { }, "status": "Failure", "message": "forbidden: User "system:anonymous" cannot get path "/"", "reason": "Forbidden", "details": { }, "code": 403 } I am total newbee and anything which is not clear into documentation I can't resolve. I am using this for testing purposes and stil after 10days can't get dashboard nice and clean and then explore full power of kubernetes. Right now it is unusable even for testing.
Hi All, Need help here current i'm installing Kubernetes version: v1.8.0 using the command "kubeadm reset && kubeadm init" it went hanged state. systemctl status kubelet log showing that Oct 04 20:15:02 cluster-1 kubelet[]: W1004 20:15:02. ] Unable to update cni config: No networks found in /etc/cni/net.d Oct 04 20:15:02 cluster-1 kubelet[]: E1004 20:15:02. ] Container runtime network not ready: NetworkReady=false reason:Net...nitialized Oct 04 20:15:02 cluster-1 kubelet[]: E1004 20:15:02. ] Failed to get system container stats for "" Oct 04 20:15:02 cluster-1 kubelet[]: E1004 20:15:02. ] Failed to get system container stats for "" Oct 04 20:15:07 cluster-1 kubelet[]: W1004 20:15:07. ] Unable to update cni config: No networks found in /etc/cni/net.d Oct 04 20:15:07 cluster-1 kubelet[]: E1004 20:15:07. ] Container runtime network not ready: NetworkReady=false reason:Net...nitialized Oct 04 20:15:12 cluster-1 kubelet[]: W1004 20:15:12. ] Unable to update cni config: No networks found in /etc/cni/net.d Oct 04 20:15:12 cluster-1 kubelet[]: E1004 20:15:12. ] Container runtime network not ready: NetworkReady=false reason:Net...nitialized Oct 04 20:15:12 cluster-1 kubelet[]: E1004 20:15:12. ] Failed to get system container stats for "" Oct 04 20:15:12 cluster-1 kubelet[]: E1004 20:15:12. ] Failed to get system container stats for "" Warning: kubelet.service changed on disk. Run 'systemctl daemon-reload' to reload units. After that i'm tried to weave it failed with error kubectl apply -f "$(kubectl version tr -d 'n')" The connection to the server localhost:8080 was refused - did you specify the right host or port? W1004 18:53:33. ] Failed to download OpenAPI (Get http://localhost:8080/swagger- dial tcp [::1]:8080: getsockopt: connection refused), falling back to swagger The connection to the server localhost:8080 was refused - did you specify the right host or port? Please anyone can help me on this
be sure to follow the instructions to add the admin kubeconfig file so kubectl can communicate with the initialized API server:
For anyone following this thread, if you encounter issues other than the message, please report them in a separate issue (after searching for existing reports), to ensure they are triaged and routed appropriately and resolved as quickly as possible. Thanks.
thank you for your quick response now i'm getting kubectl apply -f "$(kubectl version tr -d 'n')" Unable to connect to the server: Forbidden W1004 20:34:18. ] Failed to download OpenAPI (Get https://192.168.1.3:6443/swagger- Forbidden), falling back to swagger Unable to connect to the server: Forbidden
you seem to be hitting a totally different problem than the one related to ; can you please open a different issue? Thank you so much!
I also the same issue'var/lib/kubelet is not empty' after I used the command "kubeadm reset && kubeadm init" it want to hanged state.(if want u can see my above threads).to move to another step I'm trying to install the weave it was failed saying that kubectl apply -f "$(kubectl version tr -d 'n')" The connection to the server localhost:8080 was refused - did you specify the right host or port? W1004 18:53:33. ] Failed to download OpenAPI (Get http://localhost:8080/swagger- dial tcp [::1]:8080: getsockopt: connection refused), falling back to swagger The connection to the server localhost:8080 was refused - did you specify the right host or port? Can you please help me to solve the issue Thank in advance
pick to 1.8 branch is reopening to track issue until resolved in 1.8.x
if you get it means that your kube config file is missing. By default, tries to connect to . But when you deploy a cluster with , the API will be listening on port 6443. So you should see an error relative to port 6443, nor port 8080. If you see an error related to port 8080, it means that you haven't copied the configuration file () generated by . I hope that makes sense. My piece of advice would be to destroy the machines that you are currently using and restart from scratch (assuming you are using VMs), making sure that you follow each step carefully. Very often it solved the problem for me, because I had forgotten a step, or deviated from the instructions. Especially if something fails, it is often easier to restart from scratch, until you understand fully the technology and each underlying command. Good luck!
Hello hemaprasad, The problem is that when people know it is hard for them to set their mind to explain things to people who are new in a same area. Explanations given by jpetazzo and liggitt are ok but they are stepping over some steps which they assume but we as newbee don't. Now, to have kubernetes operational we all should: -stop the kubelet service, remove /var/lib/kubelet/pki, then run kubeadm init (without starting kubelet service first) Same goes for nodes: -stop the kubelet service, remove /var/lib/kubelet/pki, then run kubeadm join (with a token saved from kubeadm init) Now... how to make dashboard accessible from my workstation which is out of kubernetes cluster? Suppose for this I should go to some other thread?
your explanations are correct! As for your dashboard question, I have no idea, but I would recommend to open another github issue for that (since the title of this issue is "kubeadm 1.8.0 init fails with "/var/lib/kubelet is not empty""). Thank you!
Thank you for fast reply. Also I am trying these innocent commands from documentations: scp root . ./kubectl --kubeconfig get nodes but I am getting: Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes") My guess is that is also the same reason why dashboard over proxy is not working. Some help?
It looks like you are copying from a remote host. If it is the case, you also need to edit to point to that remote host (instead of ). The tutorials and documentations imply that people will run directly from the master.
Nice this is... so why into same documentation exist this: Controlling your cluster from machines other than the master In order to get a kubectl on some other computer (e.g. laptop) to talk to your cluster, you need to copy the administrator kubeconfig file from your master to your workstation like this... Never mind, it is obviously long way to go till this function even in test environment :(
Hello all. Finally dashboard starts working from master node. Please, do NOT use 1GB-2GB on master node, it would behave erratically and services which you expect to start normally... won't. Use at least 4GB and then also be patient. http://localhost:8001/ui/ will respond with strange formed URL so please use copy-paste: http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ That's for today. Hope this will help somebody.
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I've tried you're workaround But I'm then on kubeadm init I'm hitting this:
this is fixed in v1.8.1
After installing 1.8.1 kubelet won't start:
Not only what you mentioned dkirrane, there is a problem with: They never wrote ANY detail how to add a new Node in a Cluster when your initial Token (which is valid for ... unknown) expired! Not a word. Only if you want to test-and-try yourself. Lovely beta troubleshooting job this is. Documentation should be much, much better. Especially for somebody not working on Kubernetes dev. team.
If you try with this you will see that there is no way to use IP address mentioned: (Text is from: ) "To implement this automation, you must know the IP address that the master will have after it is started..... kubeadm can generate a token for you: kubeadm token generate Start both the master node and the worker nodes concurrently with this token." Where to use mentioned IP address? How to make any useful Node cluster joining after Token has expired?
Hello, solved a Token problem. Documentation really needs to be updated considering newbee mind. Otherwise it is source of frustration. Currently nodes are ok and operational. If this above helps you... ok.
This issue is fixed in the latest version of kubernetes. Just reset the kubeadm and remove the /var/lib/etcd as kubeadm is expecting it to be empty. Then start the kubeadm init. It should work fine.

Related discussion:
text: RequireKubeConfig:       false, KubeConfig:              flag.NewStringFlag("/var/lib/kubelet/kubeconfig"), ContainerRuntimeOptions: *NewContainerRuntimeOptions(), <del> CertDirectory:           "/var/run/kubernetes", </del> <ins> CertDirectory:           "/var/lib/kubelet/pki", </ins> RootDirectory:           v1alpha1.DefaultRootDir, // DEPRECATED: auto detecting cloud providers goes against the initiative // for out-of-tree cloud providers as we'll now depend on cAdvisor integrations