Support Red Hat OpenShift Service on AWS 4 Red Hat OpenShift Service on AWS Support. Red Hat OpenShift Documentation Team

Commands:
oc api-resources -o name | grep config.openshift.io
oc explain <resource_name>.config.openshift.io
oc get <resource_name>.config -o yaml
oc edit <resource_name>.config -o yaml
oc get clusterversion -o jsonpath='{.items[].spec.clusterID}{"\n"}'
curl -G -k -H "Authorization: Bearer USD(oc whoami -t)" https://USD(oc get route prometheus-k8s-federate -n openshift-monitoring -o jsonpath="{.spec.host}")/federate --data-urlencode 'match[]={__name__=~"cluster:usage:.*"}' --data-urlencode 'match[]={__name__="count:up0"}' --data-urlencode 'match[]={__name__="count:up1"}' --data-urlencode 'match[]={__name__="cluster_version"}' --data-urlencode 'match[]={__name__="cluster_version_available_updates"}' --data-urlencode 'match[]={__name__="cluster_version_capability"}' --data-urlencode 'match[]={__name__="cluster_operator_up"}' --data-urlencode 'match[]={__name__="cluster_operator_conditions"}' --data-urlencode 'match[]={__name__="cluster_version_payload"}' --data-urlencode 'match[]={__name__="cluster_installer"}' --data-urlencode 'match[]={__name__="cluster_infrastructure_provider"}' --data-urlencode 'match[]={__name__="cluster_feature_set"}' --data-urlencode 'match[]={__name__="instance:etcd_object_counts:sum"}' --data-urlencode 'match[]={__name__="ALERTS",alertstate="firing"}' --data-urlencode 'match[]={__name__="code:apiserver_request_total:rate:sum"}' --data-urlencode 'match[]={__name__="cluster:capacity_cpu_cores:sum"}' --data-urlencode 'match[]={__name__="cluster:capacity_memory_bytes:sum"}' --data-urlencode 'match[]={__name__="cluster:cpu_usage_cores:sum"}' --data-urlencode 'match[]={__name__="cluster:memory_usage_bytes:sum"}' --data-urlencode 'match[]={__name__="openshift:cpu_usage_cores:sum"}' --data-urlencode 'match[]={__name__="openshift:memory_usage_bytes:sum"}' --data-urlencode 'match[]={__name__="workload:cpu_usage_cores:sum"}' --data-urlencode 'match[]={__name__="workload:memory_usage_bytes:sum"}' --data-urlencode 'match[]={__name__="cluster:virt_platform_nodes:sum"}' --data-urlencode 'match[]={__name__="cluster:node_instance_type_count:sum"}' --data-urlencode 'match[]={__name__="cnv:vmi_status_running:count"}' --data-urlencode 'match[]={__name__="cluster:vmi_request_cpu_cores:sum"}' --data-urlencode 'match[]={__name__="node_role_os_version_machine:cpu_capacity_cores:sum"}' --data-urlencode 'match[]={__name__="node_role_os_version_machine:cpu_capacity_sockets:sum"}' --data-urlencode 'match[]={__name__="subscription_sync_total"}' --data-urlencode 'match[]={__name__="olm_resolution_duration_seconds"}' --data-urlencode 'match[]={__name__="csv_succeeded"}' --data-urlencode 'match[]={__name__="csv_abnormal"}' --data-urlencode 'match[]={__name__="cluster:kube_persistentvolumeclaim_resource_requests_storage_bytes:provisioner:sum"}' --data-urlencode 'match[]={__name__="cluster:kubelet_volume_stats_used_bytes:provisioner:sum"}' --data-urlencode 'match[]={__name__="ceph_cluster_total_bytes"}' --data-urlencode 'match[]={__name__="ceph_cluster_total_used_raw_bytes"}' --data-urlencode 'match[]={__name__="ceph_health_status"}' --data-urlencode 'match[]={__name__="odf_system_raw_capacity_total_bytes"}' --data-urlencode 'match[]={__name__="odf_system_raw_capacity_used_bytes"}' --data-urlencode 'match[]={__name__="odf_system_health_status"}' --data-urlencode 'match[]={__name__="job:ceph_osd_metadata:count"}' --data-urlencode 'match[]={__name__="job:kube_pv:count"}' --data-urlencode 'match[]={__name__="job:odf_system_pvs:count"}' --data-urlencode 'match[]={__name__="job:ceph_pools_iops:total"}' --data-urlencode 'match[]={__name__="job:ceph_pools_iops_bytes:total"}' --data-urlencode 'match[]={__name__="job:ceph_versions_running:count"}' --data-urlencode 'match[]={__name__="job:noobaa_total_unhealthy_buckets:sum"}' --data-urlencode 'match[]={__name__="job:noobaa_bucket_count:sum"}' --data-urlencode 'match[]={__name__="job:noobaa_total_object_count:sum"}' --data-urlencode 'match[]={__name__="odf_system_bucket_count", system_type="OCS", system_vendor="Red Hat"}' --data-urlencode 'match[]={__name__="odf_system_objects_total", system_type="OCS", system_vendor="Red Hat"}' --data-urlencode 'match[]={__name__="noobaa_accounts_num"}' --data-urlencode 'match[]={__name__="noobaa_total_usage"}' --data-urlencode 'match[]={__name__="console_url"}' --data-urlencode 'match[]={__name__="cluster:ovnkube_master_egress_routing_via_host:max"}' --data-urlencode 'match[]={__name__="cluster:network_attachment_definition_instances:max"}' --data-urlencode 'match[]={__name__="cluster:network_attachment_definition_enabled_instance_up:max"}' --data-urlencode 'match[]={__name__="cluster:ingress_controller_aws_nlb_active:sum"}' --data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:min"}' --data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:max"}' --data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:avg"}' --data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:median"}' --data-urlencode 'match[]={__name__="cluster:openshift_route_info:tls_termination:sum"}' --data-urlencode 'match[]={__name__="insightsclient_request_send_total"}' --data-urlencode 'match[]={__name__="cam_app_workload_migrations"}' --data-urlencode 'match[]={__name__="cluster:apiserver_current_inflight_requests:sum:max_over_time:2m"}' --data-urlencode 'match[]={__name__="cluster:alertmanager_integrations:max"}' --data-urlencode 'match[]={__name__="cluster:telemetry_selected_series:count"}' --data-urlencode 'match[]={__name__="openshift:prometheus_tsdb_head_series:sum"}' --data-urlencode 'match[]={__name__="openshift:prometheus_tsdb_head_samples_appended_total:sum"}' --data-urlencode 'match[]={__name__="monitoring:container_memory_working_set_bytes:sum"}' --data-urlencode 'match[]={__name__="namespace_job:scrape_series_added:topk3_sum1h"}' --data-urlencode 'match[]={__name__="namespace_job:scrape_samples_post_metric_relabeling:topk3"}' --data-urlencode 'match[]={__name__="monitoring:haproxy_server_http_responses_total:sum"}' --data-urlencode 'match[]={__name__="rhmi_status"}' --data-urlencode 'match[]={__name__="status:upgrading:version:rhoam_state:max"}' --data-urlencode 'match[]={__name__="state:rhoam_critical_alerts:max"}' --data-urlencode 'match[]={__name__="state:rhoam_warning_alerts:max"}' --data-urlencode 'match[]={__name__="rhoam_7d_slo_percentile:max"}' --data-urlencode 'match[]={__name__="rhoam_7d_slo_remaining_error_budget:max"}' --data-urlencode 'match[]={__name__="cluster_legacy_scheduler_policy"}' --data-urlencode 'match[]={__name__="cluster_master_schedulable"}' --data-urlencode 'match[]={__name__="che_workspace_status"}' --data-urlencode 'match[]={__name__="che_workspace_started_total"}' --data-urlencode 'match[]={__name__="che_workspace_failure_total"}' --data-urlencode 'match[]={__name__="che_workspace_start_time_seconds_sum"}' --data-urlencode 'match[]={__name__="che_workspace_start_time_seconds_count"}' --data-urlencode 'match[]={__name__="cco_credentials_mode"}' --data-urlencode 'match[]={__name__="cluster:kube_persistentvolume_plugin_type_counts:sum"}' --data-urlencode 'match[]={__name__="visual_web_terminal_sessions_total"}' --data-urlencode 'match[]={__name__="acm_managed_cluster_info"}' --data-urlencode 'match[]={__name__="cluster:vsphere_vcenter_info:sum"}' --data-urlencode 'match[]={__name__="cluster:vsphere_esxi_version_total:sum"}' --data-urlencode 'match[]={__name__="cluster:vsphere_node_hw_version_total:sum"}' --data-urlencode 'match[]={__name__="openshift:build_by_strategy:sum"}' --data-urlencode 'match[]={__name__="rhods_aggregate_availability"}' --data-urlencode 'match[]={__name__="rhods_total_users"}' --data-urlencode 'match[]={__name__="instance:etcd_disk_wal_fsync_duration_seconds:histogram_quantile",quantile="0.99"}' --data-urlencode 'match[]={__name__="instance:etcd_mvcc_db_total_size_in_bytes:sum"}' --data-urlencode 'match[]={__name__="instance:etcd_network_peer_round_trip_time_seconds:histogram_quantile",quantile="0.99"}' --data-urlencode 'match[]={__name__="instance:etcd_mvcc_db_total_size_in_use_in_bytes:sum"}' --data-urlencode 'match[]={__name__="instance:etcd_disk_backend_commit_duration_seconds:histogram_quantile",quantile="0.99"}' --data-urlencode 'match[]={__name__="jaeger_operator_instances_storage_types"}' --data-urlencode 'match[]={__name__="jaeger_operator_instances_strategies"}' --data-urlencode 'match[]={__name__="jaeger_operator_instances_agent_strategies"}' --data-urlencode 'match[]={__name__="appsvcs:cores_by_product:sum"}' --data-urlencode 'match[]={__name__="nto_custom_profiles:count"}' --data-urlencode 'match[]={__name__="openshift_csi_share_configmap"}' --data-urlencode 'match[]={__name__="openshift_csi_share_secret"}' --data-urlencode 'match[]={__name__="openshift_csi_share_mount_failures_total"}' --data-urlencode 'match[]={__name__="openshift_csi_share_mount_requests_total"}' --data-urlencode 'match[]={__name__="cluster:velero_backup_total:max"}' --data-urlencode 'match[]={__name__="cluster:velero_restore_total:max"}' --data-urlencode 'match[]={__name__="eo_es_storage_info"}' --data-urlencode 'match[]={__name__="eo_es_redundancy_policy_info"}' --data-urlencode 'match[]={__name__="eo_es_defined_delete_namespaces_total"}' --data-urlencode 'match[]={__name__="eo_es_misconfigured_memory_resources_info"}' --data-urlencode 'match[]={__name__="cluster:eo_es_data_nodes_total:max"}' --data-urlencode 'match[]={__name__="cluster:eo_es_documents_created_total:sum"}' --data-urlencode 'match[]={__name__="cluster:eo_es_documents_deleted_total:sum"}' --data-urlencode 'match[]={__name__="pod:eo_es_shards_total:max"}' --data-urlencode 'match[]={__name__="eo_es_cluster_management_state_info"}' --data-urlencode 'match[]={__name__="imageregistry:imagestreamtags_count:sum"}' --data-urlencode 'match[]={__name__="imageregistry:operations_count:sum"}' --data-urlencode 'match[]={__name__="log_logging_info"}' --data-urlencode 'match[]={__name__="log_collector_error_count_total"}' --data-urlencode 'match[]={__name__="log_forwarder_pipeline_info"}' --data-urlencode 'match[]={__name__="log_forwarder_input_info"}' --data-urlencode 'match[]={__name__="log_forwarder_output_info"}' --data-urlencode 'match[]={__name__="cluster:log_collected_bytes_total:sum"}' --data-urlencode 'match[]={__name__="cluster:log_logged_bytes_total:sum"}' --data-urlencode 'match[]={__name__="cluster:kata_monitor_running_shim_count:sum"}' --data-urlencode 'match[]={__name__="platform:hypershift_hostedclusters:max"}' --data-urlencode 'match[]={__name__="platform:hypershift_nodepools:max"}' --data-urlencode 'match[]={__name__="namespace:noobaa_unhealthy_bucket_claims:max"}' --data-urlencode 'match[]={__name__="namespace:noobaa_buckets_claims:max"}' --data-urlencode 'match[]={__name__="namespace:noobaa_unhealthy_namespace_resources:max"}' --data-urlencode 'match[]={__name__="namespace:noobaa_namespace_resources:max"}' --data-urlencode 'match[]={__name__="namespace:noobaa_unhealthy_namespace_buckets:max"}' --data-urlencode 'match[]={__name__="namespace:noobaa_namespace_buckets:max"}' --data-urlencode 'match[]={__name__="namespace:noobaa_accounts:max"}' --data-urlencode 'match[]={__name__="namespace:noobaa_usage:max"}' --data-urlencode 'match[]={__name__="namespace:noobaa_system_health_status:max"}' --data-urlencode 'match[]={__name__="ocs_advanced_feature_usage"}' --data-urlencode 'match[]={__name__="os_image_url_override:sum"}' --data-urlencode 'match[]={__name__="openshift:openshift_network_operator_ipsec_state:info"}'
apiVersion: v1 kind: ConfigMap data: config.yaml: | dataReporting: obfuscation: - workload_names
oc adm must-gather --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.18.0
oc adm must-gather -- /usr/bin/gather_audit_logs
NAMESPACE NAME READY STATUS RESTARTS AGE openshift-must-gather-5drcj must-gather-bklx4 2/2 Running 0 72s openshift-must-gather-5drcj must-gather-s8sdh 2/2 Running 0 72s
oc adm must-gather --run-namespace <namespace> --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.18.0
oc adm must-gather
tar cvaf must-gather.tar.gz must-gather.local.5421342344627712289/ 1
oc adm must-gather --image-stream=openshift/must-gather \ 1 --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.18.0 2
oc adm must-gather --image=USD(oc -n openshift-logging get deployment.apps/cluster-logging-operator -o jsonpath='{.spec.template.spec.containers[?(@.name == "cluster-logging-operator")].image}')
├── cluster-logging │ ├── clo │ │ ├── cluster-logging-operator-74dd5994f-6ttgt │ │ ├── clusterlogforwarder_cr │ │ ├── cr │ │ ├── csv │ │ ├── deployment │ │ └── logforwarding_cr │ ├── collector │ │ ├── fluentd-2tr64 │ ├── eo │ │ ├── csv │ │ ├── deployment │ │ └── elasticsearch-operator-7dc7d97b9d-jb4r4 │ ├── es │ │ ├── cluster-elasticsearch │ │ │ ├── aliases │ │ │ ├── health │ │ │ ├── indices │ │ │ ├── latest_documents.json │ │ │ ├── nodes │ │ │ ├── nodes_stats.json │ │ │ └── thread_pool │ │ ├── cr │ │ ├── elasticsearch-cdm-lp8l38m0-1-794d6dd989-4jxms │ │ └── logs │ │ ├── elasticsearch-cdm-lp8l38m0-1-794d6dd989-4jxms │ ├── install │ │ ├── co_logs │ │ ├── install_plan │ │ ├── olmo_logs │ │ └── subscription │ └── kibana │ ├── cr │ ├── kibana-9d69668d4-2rkvz ├── cluster-scoped-resources │ └── core │ ├── nodes │ │ ├── ip-10-0-146-180.eu-west-1.compute.internal.yaml │ └── persistentvolumes │ ├── pvc-0a8d65d9-54aa-4c44-9ecc-33d9381e41c1.yaml ├── event-filter.html ├── gather-debug.log └── namespaces ├── openshift-logging │ ├── apps │ │ ├── daemonsets.yaml │ │ ├── deployments.yaml │ │ ├── replicasets.yaml │ │ └── statefulsets.yaml │ ├── batch │ │ ├── cronjobs.yaml │ │ └── jobs.yaml │ ├── core │ │ ├── configmaps.yaml │ │ ├── endpoints.yaml │ │ ├── events │ │ │ ├── elasticsearch-im-app-1596020400-gm6nl.1626341a296c16a1.yaml │ │ │ ├── elasticsearch-im-audit-1596020400-9l9n4.1626341a2af81bbd.yaml │ │ │ ├── elasticsearch-im-infra-1596020400-v98tk.1626341a2d821069.yaml │ │ │ ├── elasticsearch-im-app-1596020400-cc5vc.1626341a3019b238.yaml │ │ │ ├── elasticsearch-im-audit-1596020400-s8d5s.1626341a31f7b315.yaml │ │ │ ├── elasticsearch-im-infra-1596020400-7mgv8.1626341a35ea59ed.yaml │ │ ├── events.yaml │ │ ├── persistentvolumeclaims.yaml │ │ ├── pods.yaml │ │ ├── replicationcontrollers.yaml │ │ ├── secrets.yaml │ │ └── services.yaml │ ├── openshift-logging.yaml │ ├── pods │ │ ├── cluster-logging-operator-74dd5994f-6ttgt │ │ │ ├── cluster-logging-operator │ │ │ │ └── cluster-logging-operator │ │ │ │ └── logs │ │ │ │ ├── current.log │ │ │ │ ├── previous.insecure.log │ │ │ │ └── previous.log │ │ │ └── cluster-logging-operator-74dd5994f-6ttgt.yaml │ │ ├── cluster-logging-operator-registry-6df49d7d4-mxxff │ │ │ ├── cluster-logging-operator-registry │ │ │ │ └── cluster-logging-operator-registry │ │ │ │ └── logs │ │ │ │ ├── current.log │ │ │ │ ├── previous.insecure.log │ │ │ │ └── previous.log │ │ │ ├── cluster-logging-operator-registry-6df49d7d4-mxxff.yaml │ │ │ └── mutate-csv-and-generate-sqlite-db │ │ │ └── mutate-csv-and-generate-sqlite-db │ │ │ └── logs │ │ │ ├── current.log │ │ │ ├── previous.insecure.log │ │ │ └── previous.log │ │ ├── elasticsearch-cdm-lp8l38m0-1-794d6dd989-4jxms │ │ ├── elasticsearch-im-app-1596030300-bpgcx │ │ │ ├── elasticsearch-im-app-1596030300-bpgcx.yaml │ │ │ └── indexmanagement │ │ │ └── indexmanagement │ │ │ └── logs │ │ │ ├── current.log │ │ │ ├── previous.insecure.log │ │ │ └── previous.log │ │ ├── fluentd-2tr64 │ │ │ ├── fluentd │ │ │ │ └── fluentd │ │ │ │ └── logs │ │ │ │ ├── current.log │ │ │ │ ├── previous.insecure.log │ │ │ │ └── previous.log │ │ │ ├── fluentd-2tr64.yaml │ │ │ └── fluentd-init │ │ │ └── fluentd-init │ │ │ └── logs │ │ │ ├── current.log │ │ │ ├── previous.insecure.log │ │ │ └── previous.log │ │ ├── kibana-9d69668d4-2rkvz │ │ │ ├── kibana │ │ │ │ └── kibana │ │ │ │ └── logs │ │ │ │ ├── current.log │ │ │ │ ├── previous.insecure.log │ │ │ │ └── previous.log │ │ │ ├── kibana-9d69668d4-2rkvz.yaml │ │ │ └── kibana-proxy │ │ │ └── kibana-proxy │ │ │ └── logs │ │ │ ├── current.log │ │ │ ├── previous.insecure.log │ │ │ └── previous.log │ └── route.openshift.io │ └── routes.yaml └── openshift-operators-redhat ├──
oc adm must-gather --image-stream=openshift/must-gather \ 1 --image=quay.io/kubevirt/must-gather 2
tar cvaf must-gather.tar.gz must-gather.local.5421342344627712289/ 1
oc adm must-gather -- gather_network_logs
tar cvaf must-gather.tar.gz must-gather.local.472290403699006248 1
Disk usage exceeds the volume percentage of 30% for mounted directory. Exiting
oc adm must-gather --volume-percentage <storage_percentage>
oc get clusterversion -o jsonpath='{.items[].spec.clusterID}{"\n"}'
oc adm node-logs --role=master -u kubelet 1
oc adm node-logs --role=master --path=openshift-apiserver
oc adm node-logs --role=master --path=openshift-apiserver/audit.log
oc adm must-gather --dest-dir /tmp/captures \// <.> --source-dir '/tmp/tcpdump/' \// <.> --image registry.redhat.io/openshift4/network-tools-rhel8:latest \// <.> --node-selector 'node-role.kubernetes.io/worker' \// <.> --host-network=true \// <.> --timeout 30s \// <.> -- tcpdump -i any \// <.> -w /tmp/tcpdump/%Y-%m-%dT%H:%M:%S.pcap -W 1 -G 300
tmp/captures ├── event-filter.html ├── ip-10-0-192-217-ec2-internal 1 │ └── registry-redhat-io-openshift4-network-tools-rhel8-sha256-bca │ └── 2022-01-13T19:31:31.pcap ├── ip-10-0-201-178-ec2-internal 2 │ └── registry-redhat-io-openshift4-network-tools-rhel8-sha256-bca │ └── 2022-01-13T19:31:30.pcap ├── ip- └── timestamp
oc get nodes
oc debug node/my-cluster-node
chroot /host
ip ad
toolbox
tcpdump -nn -s 0 -i ens5 -w /host/var/tmp/my-cluster-node_USD(date +%d_%m_%Y-%H_%M_%S-%Z).pcap 1
chroot /host crictl ps
chroot /host crictl inspect --output yaml a7fe32346b120 | grep 'pid' | awk '{print USD2}'
nsenter -n -t 49628 -- tcpdump -nn -i ens5 -w /host/var/tmp/my-cluster-node-my-container_USD(date +%d_%m_%Y-%H_%M_%S-%Z).pcap 1
oc debug node/my-cluster-node -- bash -c 'cat /host/var/tmp/my-tcpdump-capture-file.pcap' > /tmp/my-tcpdump-capture-file.pcap 1
oc debug node/my-cluster-node -- bash -c 'cat /host/var/tmp/my-diagnostic-data.tar.gz' > /var/tmp/my-diagnostic-data.tar.gz 1
chroot /host
toolbox
dnf install -y <package_name>
chroot /host
REGISTRY=quay.io 1 IMAGE=fedora/fedora:latest 2 TOOLBOX_NAME=toolbox-fedora-latest 3
toolbox
oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.13.8 True False 8h Cluster version is 4.13.8
oc describe clusterversion
Name: version Namespace: Labels: <none> Annotations: <none> API Version: config.openshift.io/v1 Kind: ClusterVersion Image: quay.io/openshift-release-dev/ocp-release@sha256:a956488d295fe5a59c8663a4d9992b9b5d0950f510a7387dbbfb8d20fc5970ce URL: https://access.redhat.com/errata/RHSA-2023:4456 Version: 4.13.8 History: Completion Time: 2023-08-17T13:20:21Z Image: quay.io/openshift-release-dev/ocp-release@sha256:a956488d295fe5a59c8663a4d9992b9b5d0950f510a7387dbbfb8d20fc5970ce Started Time: 2023-08-17T12:59:45Z State: Completed Verified: false Version: 4.13.8
rosa logs install --cluster=<cluster_name>
rosa logs install --cluster=<cluster_name> --watch
rosa logs uninstall --cluster=<cluster_name>
rosa logs uninstall --cluster=<cluster_name> --watch
rosa verify permissions
rosa verify quota
oc get nodes
oc adm top nodes
oc adm top node my-node
oc get subs -n <operator_namespace>
oc describe sub <subscription_name> -n <operator_namespace>
Name: cluster-logging Namespace: openshift-logging Labels: operators.coreos.com/cluster-logging.openshift-logging= Annotations: <none> API Version: operators.coreos.com/v1alpha1 Kind: Subscription Conditions: Last Transition Time: 2019-07-29T13:42:57Z Message: all available catalogsources are healthy Reason: AllCatalogSourcesHealthy Status: False Type: CatalogSourcesUnhealthy
oc get catalogsources -n openshift-marketplace
NAME DISPLAY TYPE PUBLISHER AGE certified-operators Certified Operators grpc Red Hat 55m community-operators Community Operators grpc Red Hat 55m example-catalog Example Catalog grpc Example Org 2m25s redhat-marketplace Red Hat Marketplace grpc Red Hat 55m redhat-operators Red Hat Operators grpc Red Hat 55m
oc describe catalogsource example-catalog -n openshift-marketplace
Name: example-catalog Namespace: openshift-marketplace Labels: <none> Annotations: operatorframework.io/managed-by: marketplace-operator target.workload.openshift.io/management: {"effect": "PreferredDuringScheduling"} API Version: operators.coreos.com/v1alpha1 Kind: CatalogSource Status: Connection State: Address: example-catalog.openshift-marketplace.svc:50051 Last Connect: 2021-09-09T17:07:35Z Last Observed State: TRANSIENT_FAILURE Registry Service: Created At: 2021-09-09T17:05:45Z Port: 50051 Protocol: grpc Service Name: example-catalog Service Namespace: openshift-marketplace
oc get pods -n openshift-marketplace
NAME READY STATUS RESTARTS AGE certified-operators-cv9nn 1/1 Running 0 36m community-operators-6v8lp 1/1 Running 0 36m marketplace-operator-86bfc75f9b-jkgbc 1/1 Running 0 42m example-catalog-bwt8z 0/1 ImagePullBackOff 0 3m55s redhat-marketplace-57p8c 1/1 Running 0 36m redhat-operators-smxx8 1/1 Running 0 36m
oc describe pod example-catalog-bwt8z -n openshift-marketplace
Name: example-catalog-bwt8z Namespace: openshift-marketplace Priority: 0 Node: ci-ln-jyryyg2-f76d1-ggdbq-worker-b-vsxjd/10.0.128.2 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 48s default-scheduler Successfully assigned openshift-marketplace/example-catalog-bwt8z to ci-ln-jyryyf2-f76d1-fgdbq-worker-b-vsxjd Normal AddedInterface 47s multus Add eth0 [10.131.0.40/23] from openshift-sdn Normal BackOff 20s (x2 over 46s) kubelet Back-off pulling image "quay.io/example-org/example-catalog:v1" Warning Failed 20s (x2 over 46s) kubelet Error: ImagePullBackOff Normal Pulling 8s (x3 over 47s) kubelet Pulling image "quay.io/example-org/example-catalog:v1" Warning Failed 8s (x3 over 47s) kubelet Failed to pull image "quay.io/example-org/example-catalog:v1": rpc error: code = Unknown desc = reading manifest v1 in quay.io/example-org/example-catalog: unauthorized: access to the requested resource is not authorized Warning Failed 8s (x3 over 47s) kubelet Error: ErrImagePull
oc get clusteroperators
oc get pod -n <operator_namespace>
oc describe pod <operator_pod_name> -n <operator_namespace>
oc get pods -n <operator_namespace>
oc logs pod/<pod_name> -n <operator_namespace>
oc logs pod/<operator_pod_name> -c <container_name> -n <operator_namespace>
ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl pods
ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl inspectp <operator_pod_id>
ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl ps --pod=<operator_pod_id>
ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl inspect <container_id>
ssh core@<master-node>.<cluster_name>.<base_domain> sudo crictl logs -f <container_id>
oc project <project_name>
oc get pods
oc status
skopeo inspect docker://<image_reference>
oc edit deployment/my-deployment
oc get pods -w
oc get events
oc logs <pod_name>
oc logs <pod_name> -c <container_name>
oc exec <pod_name> -- ls -alh /var/log
total 124K drwxr-xr-x. 1 root root 33 Aug 11 11:23 . drwxr-xr-x. 1 root root 28 Sep 6 2022 .. -rw-rw----. 1 root utmp 0 Jul 10 10:31 btmp -rw-r--r--. 1 root root 33K Jul 17 10:07 dnf.librepo.log -rw-r--r--. 1 root root 69K Jul 17 10:07 dnf.log -rw-r--r--. 1 root root 8.8K Jul 17 10:07 dnf.rpm.log -rw-r--r--. 1 root root 480 Jul 17 10:07 hawkey.log -rw-rw-r--. 1 root utmp 0 Jul 10 10:31 lastlog drwx------. 2 root root 23 Aug 11 11:14 openshift-apiserver drwx------. 2 root root 6 Jul 10 10:31 private drwxr-xr-x. 1 root root 22 Mar 9 08:05 rhsm -rw-rw-r--. 1 root utmp 0 Jul 10 10:31 wtmp
oc exec <pod_name> cat /var/log/<path_to_log>
2023-07-10T10:29:38+0000 INFO --- logging initialized --- 2023-07-10T10:29:38+0000 DDEBUG timer: config: 13 ms 2023-07-10T10:29:38+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile 2023-07-10T10:29:38+0000 INFO Updating Subscription Management repositories. 2023-07-10T10:29:38+0000 INFO Unable to read consumer identity 2023-07-10T10:29:38+0000 INFO Subscription Manager is operating in container mode. 2023-07-10T10:29:38+0000 INFO
oc exec <pod_name> -c <container_name> ls /var/log
oc exec <pod_name> -c <container_name> cat /var/log/<path_to_log>
oc project <namespace>
oc rsh <pod_name> 1
oc rsh -c <container_name> pod/<pod_name>
oc port-forward <pod_name> <host_port>:<pod_port> 1
oc get deployment -n <project_name>
oc debug deployment/my-deployment --as-root -n <project_name>
oc get deploymentconfigs -n <project_name>
oc debug deploymentconfig/my-deployment-configuration --as-root -n <project_name>
oc cp <local_path> <pod_name>:/<path> -c <container_name> 1
oc cp <pod_name>:/<path> -c <container_name> <local_path> 1
oc get pods -w 1
oc logs -f pod/<application_name>-<build_number>-build
oc logs -f pod/<application_name>-<build_number>-deploy
oc logs -f pod/<application_name>-<build_number>-<random_string>
oc describe pod/my-app-1-akdlg
oc logs -f pod/my-app-1-akdlg
oc exec my-app-1-akdlg -- cat /var/log/my-application.log
oc debug dc/my-deployment-configuration --as-root -- cat /var/log/my-application.log
oc exec -it my-app-1-akdlg /bin/bash
Unable to attach or mount volumes: unmounted volumes=[sso-mysql-pvol], unattached volumes=[sso-mysql-pvol default-token-x4rzc]: timed out waiting for the condition Multi-Attach error for volume "pvc-8837384d-69d7-40b2-b2e6-5df86943eef9" Volume is already used by pod(s) sso-mysql-1-ns6b4
oc delete pod <old_pod> --force=true --grace-period=0
oc -n ns1 get service prometheus-example-app -o yaml
labels: app: prometheus-example-app
oc -n ns1 get servicemonitor prometheus-example-monitor -o yaml
apiVersion: v1 kind: ServiceMonitor metadata: name: prometheus-example-monitor namespace: ns1 spec: endpoints: - interval: 30s port: web scheme: http selector: matchLabels: app: prometheus-example-app
oc -n openshift-user-workload-monitoring get pods
NAME READY STATUS RESTARTS AGE prometheus-operator-776fcbbd56-2nbfm 2/2 Running 0 132m prometheus-user-workload-0 5/5 Running 1 132m prometheus-user-workload-1 5/5 Running 1 132m thanos-ruler-user-workload-0 3/3 Running 0 132m thanos-ruler-user-workload-1 3/3 Running 0 132m
oc -n openshift-user-workload-monitoring logs prometheus-operator-776fcbbd56-2nbfm -c prometheus-operator
level=warn ts=2020-08-10T11:48:20.906739623Z caller=operator.go:1829 component=prometheusoperator msg="skipping servicemonitor" error="it accesses file system via bearer token file which Prometheus specification prohibits" servicemonitor=eagle/eagle namespace=openshift-user-workload-monitoring prometheus=user-workload
oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config
apiVersion: v1 kind: ConfigMap metadata: name: user-workload-monitoring-config namespace: openshift-user-workload-monitoring data: config.yaml: | prometheusOperator: logLevel: debug
oc -n openshift-user-workload-monitoring get deploy prometheus-operator -o yaml | grep "log-level"
- --log-level=debug
oc -n openshift-user-workload-monitoring get pods
topk(10, max by(namespace, job) (topk by(namespace, job) (1, scrape_samples_post_metric_relabeling)))
topk(10, sum by(namespace, job) (sum_over_time(scrape_series_added[1h])))
HOST=USD(oc -n openshift-monitoring get route prometheus-k8s -ojsonpath='{.status.ingress[].host}')
TOKEN=USD(oc whoami -t)
curl -H "Authorization: Bearer USDTOKEN" -k "https://USDHOST/api/v1/status/tsdb"
"status": "success","data":{"headStats":{"numSeries":507473, "numLabelPairs":19832,"chunkCount":946298,"minTime":1712253600010, "maxTime":1712257935346},"seriesCountByMetricName": [{"name":"etcd_request_duration_seconds_bucket","value":51840}, {"name":"apiserver_request_sli_duration_seconds_bucket","value":47718},
oc <command> --loglevel <log_level>
oc whoami -t
sha256~RCV3Qcn7H-OEfqCGVI0CvnZ6
Can't get tokens . Can't get access tokens .
E: Failed to create cluster: The sts_user_role is not linked to account '1oNl'. Please create a user role and link it to the account.
rosa list ocm-role
I: Fetching ocm roles ROLE NAME ROLE ARN LINKED ADMIN ManagedOpenShift-OCM-Role-1158 arn:aws:iam::2066:role/ManagedOpenShift-OCM-Role-1158 No No
rosa list user-role
I: Fetching user roles ROLE NAME ROLE ARN LINKED ManagedOpenShift-User.osdocs-Role arn:aws:iam::2066:role/ManagedOpenShift-User.osdocs-Role Yes
rosa create ocm-role
rosa create ocm-role --admin
I: Creating ocm role ? Role prefix: ManagedOpenShift 1 ? Enable admin capabilities for the OCM role (optional): No 2 ? Permissions boundary ARN (optional): 3 ? Role Path (optional): 4 ? Role creation mode: auto 5 I: Creating role using 'arn:aws:iam::<ARN>:user/<UserName>' ? Create the 'ManagedOpenShift-OCM-Role-182' role? Yes 6 I: Created role 'ManagedOpenShift-OCM-Role-182' with ARN 'arn:aws:iam::<ARN>:role/ManagedOpenShift-OCM-Role-182' I: Linking OCM role ? OCM Role ARN: arn:aws:iam::<ARN>:role/ManagedOpenShift-OCM-Role-182 7 ? Link the 'arn:aws:iam::<ARN>:role/ManagedOpenShift-OCM-Role-182' role with organization '<AWS ARN>'? Yes 8 I: Successfully linked role-arn 'arn:aws:iam::<ARN>:role/ManagedOpenShift-OCM-Role-182' with organization account '<AWS ARN>'
rosa create user-role
I: Creating User role ? Role prefix: ManagedOpenShift 1 ? Permissions boundary ARN (optional): 2 ? Role Path (optional): 3 ? Role creation mode: auto 4 I: Creating ocm user role using 'arn:aws:iam::2066:user' ? Create the 'ManagedOpenShift-User.osdocs-Role' role? Yes 5 I: Created role 'ManagedOpenShift-User.osdocs-Role' with ARN 'arn:aws:iam::2066:role/ManagedOpenShift-User.osdocs-Role' I: Linking User role ? User Role ARN: arn:aws:iam::2066:role/ManagedOpenShift-User.osdocs-Role ? Link the 'arn:aws:iam::2066:role/ManagedOpenShift-User.osdocs-Role' role with account '1AGE'? Yes 6 I: Successfully linked role ARN 'arn:aws:iam::2066:role/ManagedOpenShift-User.osdocs-Role' with account '1AGE'
rosa list ocm-role
rosa list user-role
rosa link ocm-role --role-arn <arn>
I: Linking OCM role ? Link the '<AWS ACCOUNT ID>` role with organization '<ORG ID>'? Yes I: Successfully linked role-arn '<AWS ACCOUNT ID>' with organization account '<ORG ID>'
rosa link user-role --role-arn <arn>
I: Linking User role ? Link the 'arn:aws:iam::<ARN>:role/ManagedOpenShift-User-Role-125' role with organization '<AWS ID>'? Yes I: Successfully linked role-arn 'arn:aws:iam::<ARN>:role/ManagedOpenShift-User-Role-125' with organization account '<AWS ID>'
rosa create --profile <aws_profile> ocm-role
rosa create --profile <aws_profile> user-role
rosa create --profile <aws_profile> account-roles
rosa describe cluster -c <my_cluster_name> --debug
Failed to create cluster: Unable to create cluster spec: Failed to get access keys for user 'osdCcsAdmin': NoSuchEntity: The user with name osdCcsAdmin cannot be found.
rosa init --delete
rosa init
Error: Error creating network Load Balancer: AccessDenied: User: arn:aws:sts::xxxxxxxxxxxx:assumed-role/ManagedOpenShift-Installer-Role/xxxxxxxxxxxxxxxxxxx is not authorized to perform: iam:CreateServiceLinkedRole on resource: arn:aws:iam::xxxxxxxxxxxx:role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing"
aws iam get-role --role-name "AWSServiceRoleForElasticLoadBalancing" || aws iam create-service-linked-role --aws-service-name "elasticloadbalancing.amazonaws.com"
Error deleting cluster CLUSTERS-MGMT-400: Failed to delete cluster <hash>: sts_user_role is not linked to your account. sts_ocm_role is linked to your organization <org number> which requires sts_user_role to be linked to your Red Hat account <account ID>.Please create a user role and link it to the account: User Account <account ID> is not authorized to perform STS cluster operations Operation ID: b0572d6e-fe54-499b-8c97-46bf6890011c
E: Failed to delete cluster <hash>: sts_user_role is not linked to your account. sts_ocm_role is linked to your organization <org_number> which requires sts_user_role to be linked to your Red Hat account <account_id>.Please create a user role and link it to the account: User Account <account ID> is not authorized to perform STS cluster operations
rosa create user-role
I: Successfully linked role ARN <user role ARN> with account <account ID>