Change:
When a user report an issue on Kubernetes, we normally ask them to provide some outputs by running kubectl commands, such as kubectl get <resource, kubectl describe <resourceetc. We ask for daemons' log from various nodes often. Sometimes the user has to reproduce the issue for us because the config is changed. Meanwhile the developer might spend a lot of time on debugging the issues on a deprecated version or mismatched client with server. I propose to add a dump command to kubectl, which collects related outputs, logs, from various components or related nodes, then generate a tarball and attach that to the issue.
cc
This seems very useful. It could be the starting point for a more comprehensive diagnose-me that the user runs when they see a problem.
For a while now the kubelet logs have been collected by node level logging. For Elasticsearch the user needs to enable cluster level logging (on by default for Elasticsearch node level logging). For gcp node level logging the kubelet logs will have been ingested into the cloud logging console for the GCE project used to host the cluster.
cc/ They are going to help with this issue. Thanks!
I guess this issue is not closed yet (it was incorrectly closed by keyword in PR message).
Together with we propose to dump the following information: level: description of all kubernetes resources: nodes, replication controllers, services, pods: kubectl get nodes,podes,rc,services,events -o json, level: docker processes and images on each master/minion node: docker ps -a, docker images. level: version of etcd and boundpods stored in it: etcd --version, etcdctl get boundpods. (may consume a lot of space): the whole state (dump) of etcd. (may consume a lot of space): logs of the following components: kubelet, docker, etcd, kube-apiserver and kube-addons. The information will be dump by hack/kube- script. The initial version of the script (PR ) implements point 1, 2 and 3. We suggest extending the kube- by adding of the following flags: --all: dump etcd state and logs: in addition to points 1, 2 and 3, also dump information proposed in pointes 4 and 5 (this may consume a lot of space); --issue=XXX: instead of writing output to stdout, make a tarball and attach it to #XXX issue on kubernetes github.
Proposed command is (turning cluster-info into a command group).
- Is there anything else needed to be done in this PR? (can it be closed?)
The original proposal wasn't fully implemented, but a new more specific issue can be filed to record additional data.
cc/

Discussion:
<ins> #!/bin/bash # Copyright 2015 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This script dumps debugging state of currently running Kubernetes cluster. KUBE_ROOT=$(dirname "${BASH_SOURCE}")/.. echo "kube-dump.sh: Getting version..." "${KUBE_ROOT}/cluster/kubectl.sh" version echo echo "kube-dump.sh: Getting resources..." "${KUBE_ROOT}/cluster/kubectl.sh" get nodes,pods,rc,services,events -o json echo source "${KUBE_ROOT}/cluster/kube-env.sh" source "${KUBE_ROOT}/cluster/${KUBERNETES_PROVIDER}/util.sh" detect-project &> /dev/null echo "kube-dump.sh: Getting docker statuses on all nodes..." ALL_NODES=(${MINION_NAMES[*]} ${MASTER_NAME}) for NODE in ${ALL_NODES[*]}; do echo "kube-dump.sh: Node $NODE:" ssh-to-node "${NODE}" ' sudo docker ps -a sudo docker images ' done echo echo "kube-dump.sh: Getting boundpods from etcd..." ssh-to-node "${MASTER_NAME}" ' ETCD_SERVER=$(hostname -i):4001 for DIR in $(etcdctl -C $ETCD_SERVER ls /registry/nodes); do echo "kube-dump.sh: Dir $DIR:" etcdctl -C $ETCD_SERVER get $DIR/boundpods done ' </ins>