Server Outage 12-3-18 - Page 1
Forum section: electronics / general

author: EEVblog
author_rank: Unknown
timestamp: March 11, 2018, 10:36:46 pm
content: The server was down all last night. Hostgator's response attached.

author: frozenfrogz
author_rank: Frequent Contributor
timestamp: March 11, 2018, 10:44:54 pm
content: The server used to work, then it took an arrow in the knee.Nice to have it back!

author: SparkyFX
author_rank: Frequent Contributor
timestamp: March 11, 2018, 10:59:05 pm
content: Strange, but nothing an unattended dist-upgrade on certain packages couldn´t have caused.Nice to have the eevblog online again.

author: hermit
author_rank: Frequent Contributor
timestamp: March 11, 2018, 11:48:19 pm
content: I've used cPanel for years without this kind of mystery reconfiguration. @gnif has some forensic work to do from the sounds of it. Not that he won't find it was some 'unscheduled help' from the hosting company.

author: Specmaster
author_rank: Super Contributor
timestamp: March 12, 2018, 10:32:23 am
content: I notice that its not always handling quotes correctly still?

author: gnif
author_rank: Unknown
timestamp: March 12, 2018, 10:37:32 am
content: Quote from: hermit on March 11, 2018, 11:48:19 pmWhat they failed to mention was thatthey deployed the server to use DHCP, they then moved away from DHCP when they moved data centers, and never updated Dave's "managed" server, so when their DHCP server went offline, so did this (and how knows how many more) server when the DHCP lease expired.They then tried to perform a `yum update`, which failed as cPanel blacklists the perl packages. They assumed this was due to the "incompatible" CentOS repositories. Rather then contacting Dave or myself about what they wanted to do, they forcibly removed the kernel and swapped it out for a different one. Their reason for changing the kernel was valid, but the assumption that they could just do this was extremely unprofessional.Edit: I just confirmed by restoring the config from backup that it was indeed configured by HG to use DHCPCode:# cat etc/sysconfig/network-scripts/ifcfg-eth0DEVICE="eth0"BOOTPROTO="dhcp"IPV6INIT="yes"MTU="1500"NM_CONTROLLED="no"ONBOOT="yes"TYPE="Ethernet"Here is the bash history of their session and my comments.Code:501 2018-03-12 07:54:25 yum list installed | grep *redacted*502 2018-03-12 07:54:34 yum erase kernel-*redacted*503 2018-03-12 07:55:10 yum erase kernel-*redacted* kernel-*redacted*504 2018-03-12 07:55:18 yum update kernel505 2018-03-12 07:56:52 rebootNote they did not even check why it couldn't talk to the network, they just blindly assumed a kernel fault, and uninstalled itCode:506 2018-03-12 07:58:28 ping [url=http://www.goog]www.goog507 2018-03-12 07:58:33 ping [url=http://www.google.com]www.google.com508 2018-03-12 07:58:37 history509 2018-03-12 08:00:14 yum repolistOk, failed to ping google... checked bash history, not sure why. Then checked the repositories installed into yum.Code:510 2018-03-12 08:06:05 cat /etc/sysconfig/network-scripts/ifcfg-eth0Great, ok, finally looking at the network configuration. Anything else related to altering this file is missing from the history, I can only assume it was done in another terminal, or remote transfer via SCP or similar.Code:511 2018-03-12 08:10:31 for ip in $(ip a s eth0 | awk -F'[ /]' '/inet.*global/ {print $6}') ; do echo -ne "${ip}\t" ; ping -c3 -I ${ip} 8.8.8.8 2>&1 > /dev/null && echo "UP" || echo "DOWN" ; done512 2018-03-12 08:13:34 yum updateA simple script to check for connectivity and then... `yum update`The server was back online at this point... their work was done, if they wanted to do anything more they should have checked first.

author: Specmaster
author_rank: Super Contributor
timestamp: March 12, 2018, 10:49:33 am
content: Thank god we have someone who understands the system because all of that to me was just gobbledegook

author: bd139
author_rank: Super Contributor
timestamp: March 12, 2018, 10:51:21 am
content: So much experience with stuff like this it's unreal. The first thing that popped into my head when I read wilfred's post is that I bet it was the hosting company with a wrecking ball rather than "user error". What a turd!Nice to know that managed hosting companies are still as thoroughly incompetent as they were a decade ago.Also why remove DHCP? Their IPAM is probably an Excel sheet they email to each other

author: hwj-d
author_rank: Frequent Contributor
timestamp: March 12, 2018, 10:54:17 am
content: Thanks for you work. I've missed it the whole daylight sunday, which is night in australia of course without knowing what's happened. I'm glad to have you back.Small official hint, twitter and/or FB, mail would be usefull.(€: I passed through all infrastructure possibilities, ping, traceroute, dns, proxies, etc for this)

author: gnif
author_rank: Unknown
timestamp: March 12, 2018, 10:56:13 am
content: Quote from: hwj-d on March 12, 2018, 10:54:17 amYou're welcome. I can't post on Dave's Twitter when things here are down but I can post on my own if people care to follow it.https://twitter.com/HostFissionI should actually list this here too, I have a patreon for those that want to support the volunteer aspect of my work:https://www.patreon.com/gnif

author: hwj-d
author_rank: Frequent Contributor
timestamp: March 12, 2018, 11:11:30 am
content: Can't this be automated, maybe mail if the server is down more than 2h ? Dont know ...

author: EEVblog
author_rank: Unknown
timestamp: March 12, 2018, 11:16:55 am
content: Quote from: gnif on March 12, 2018, 10:56:13 am

author: bd139
author_rank: Super Contributor
timestamp: March 12, 2018, 11:19:15 am
content: Quote from: hwj-d on March 12, 2018, 11:11:30 amTake a look athttps://www.pingdom.com/... that can post on twitter if everything goes down

author: gnif
author_rank: Unknown
timestamp: March 12, 2018, 11:44:30 am
content: Quote from: bd139 on March 12, 2018, 11:19:15 amI have monitoring software setup to keep an eye on Dave's server, but it experiences ICMP packet loss so often it generates ton's of false positives, this could also be a sign of HG's network being overloaded.I did have an alert for the outage, but due to the time (1am) and the continual false alerts from this server I incorrectly assumed it was another false positive, it was when I got an email from Dave I realized it was an actual outage and acted.In short, pingdom will be plastering twitter with outage notifications when there isn't one...

author: hwj-d
author_rank: Frequent Contributor
timestamp: March 12, 2018, 11:51:33 am
content: Quote from: bd139 on March 12, 2018, 11:19:15 amOf course, I already noticed that the forum can't be reached globaly.But this doesn't replace a notification mail from server itself.I do not insist on this service and its more work. This is just a suggestion.

author: bd139
author_rank: Super Contributor
timestamp: March 12, 2018, 11:54:15 am
content: Quote from: gnif on March 12, 2018, 11:44:30 amAck you're doomed then. I wish thee luck

author: hwj-d
author_rank: Frequent Contributor
timestamp: March 12, 2018, 11:57:56 am
content: QuoteGot it. Have understanding for that.

author: Specmaster
author_rank: Super Contributor
timestamp: March 12, 2018, 11:58:59 am
content: Thank god that we don't suffer outage that much then.

author: PA0PBZ
author_rank: Super Contributor
timestamp: March 12, 2018, 12:02:38 pm
content: Quote from: hwj-d on March 12, 2018, 10:54:17 amhmm...https://twitter.com/eevblog/status/972800200199647232

author: hwj-d
author_rank: Frequent Contributor
timestamp: March 12, 2018, 12:22:28 pm
content: Quote from: PA0PBZ on March 12, 2018, 12:02:38 pmYes.Now i will follow Dave, if the server fails again.

author: station240
author_rank: Unknown
timestamp: March 12, 2018, 12:52:33 pm
content: gnif and Dave, send HostGater a bill for the repair work, with a note that says "this could have been avoided if you had actually contacted us"So many clowns in the web hosting industry, It's like a race to the bottom for who can have the cheapest plans and the least capable technical staff.

author: mnementh
author_rank: Super Contributor
timestamp: March 12, 2018, 12:57:08 pm
content: Quote from: hwj-d on March 12, 2018, 12:22:28 pmThat seems reasonable... except that many of us don't Twit because of the inverse exponential SNR. And a reasonable fear that some of the Twit-in-Chief's verbal ejaculate might get on us.mnemThanks for all the fish.

author: rrinker
author_rank: Super Contributor
timestamp: March 16, 2018, 05:37:32 pm
content: I don;t use Twitter because I though tthe whole idea was kind of stupid when it first came out.But wow, can't even get a thread on the server being down without some snarky political comment sneaking in.

author: gnif
author_rank: Unknown
timestamp: May 10, 2018, 03:35:47 am
content: Sorry about that short outage, making some server changes and missed a configuration directive.Edit: A few more caused by a stray semicolon in the puppet generated nginx config. That should be the last of it though for today

author: IanJ
author_rank: Unknown
timestamp: May 10, 2018, 06:13:30 am
content: Quote from: gnif on March 12, 2018, 10:37:32 am1and1 did that to me about a year ago on my server that I had been using for 2 years.......several calls later and several techs later they fixed it. They couldn't give me a reason why the static IP setup suddenly changed to DHCP.I've since ditched 1and1.Ian.