Wesbite & Forum Outage 10th Dec 2016 - Page 1
Forum section: electronics / general

author: EEVblog
author_rank: Unknown
timestamp: December 10, 2016, 09:31:22 am
content: Down for over 12 hours.It was the fault of Hostgator:https://forums.hostgator.com/12-09-16-15-00-multiple-t345474.htmlCue the people who say "switch to a real host" in 3, 2, 1...

author: ez24
author_rank: Super Contributor
timestamp: December 10, 2016, 09:38:04 am
content: It was tough going

author: NottheDan
author_rank: Frequent Contributor
timestamp: December 10, 2016, 09:38:26 am
content: Switch to a real gator, mateJust getting it out of the way.

author: TheSteve
author_rank: Unknown
timestamp: December 10, 2016, 09:47:19 am
content: I highly doubt they have any intention of admitting what really happened.

author: Muttley Snickers
author_rank: Unknown
timestamp: December 10, 2016, 10:21:07 am
content: I learnt how to change my nickname on the EEVblog IRC site, and then I learnt that I could join the IRC channel from my smart TV, this wouldn't have happened if the forum was up and running so not all was wasted really.Thats all I have to offer up in the way of positives.

author: SeanB
author_rank: Super Contributor
timestamp: December 10, 2016, 10:28:01 am
content: Not a problem, just went and got through a little of the YT backlog, and went to bed.

author: jonovid
author_rank: Super Contributor
timestamp: December 10, 2016, 11:03:31 am
content: QuoteDave maybe We're gonna need a bigger server

author: timb
author_rank: Super Contributor
timestamp: December 10, 2016, 12:19:46 pm
content: Jesus, I don't know how HostGator is still is business... They were a crappy, subpar host 10 years ago when I was in the hosting industry, I guess much doesn't change...Dave, if you ever decide to rent a dedicated VM on a cluster somewhere, I've got some good recommendations. It's a lot cheaper than you think! I'd also be happy to administrate it for you.

author: FrankBuss
author_rank: Unknown
timestamp: December 10, 2016, 01:45:26 pm
content: Well, why don't you switch to a real host? Did you at least negotiate unlimited traffic, as it is standard e.g. athttp://1and1.com, where my server runs (and I had never a timeout of 12 hours, just a short reboot in the last year. I recommend this for my customers, too, they would lose quite some money with 12 hour outages) ? Same athttp://www.ovh.com. Looks like many of the big providers offers this.Long time ago I had a server with max traffic and then a notification and additional costs. It got hacked and was used as a warez server for a day until I fixed it. Cost this month was multiple times the normal cost.

author: Jay_Diddy_B
author_rank: Super Contributor
timestamp: December 10, 2016, 02:33:04 pm
content: Hi group,When there is an outage like this, In my mind it just reinforces, what a great thing we have here.I have to admit that I was getting withdrawal...Regards,Jay_Diddy_B

author: Brumby
author_rank: Unknown
timestamp: December 11, 2016, 01:22:08 am
content: Aside from the separation anxiety, I noticed I knocked off a couple more items from my 'To Do' list than I expected. Luckily SWMBO didn't.

author: ez24
author_rank: Super Contributor
timestamp: December 11, 2016, 01:45:12 am
content: Is this outage a record?

author: SeanB
author_rank: Super Contributor
timestamp: December 11, 2016, 05:53:06 am
content: Quote from: ez24 on December 11, 2016, 01:45:12 amNo, there were 40 years of Dave's life when the blog was not available. Don't know how he survived that.

author: EEVblog
author_rank: Unknown
timestamp: December 15, 2016, 04:41:25 am
content: Here is Hostgators explanation:https://www.hostgator.com/blog/122016-dedicated-server-outage/

author: djos
author_rank: Unknown
timestamp: December 15, 2016, 04:49:08 am
content: Quote from: EEVblog on December 10, 2016, 09:31:22 amMaybe if you switched to a high-end hosting provider like AWS....oh wait...

author: tautech
author_rank: Super Contributor
timestamp: December 15, 2016, 04:55:44 am
content: Quote from: EEVblog on December 15, 2016, 04:41:25 amCondescending crap.

author: djos
author_rank: Unknown
timestamp: December 15, 2016, 04:58:09 am
content: Quote from: tautech on December 15, 2016, 04:55:44 amMostly yes but I don't doubt they broke their network and a spanning tree loop (or propagation) compounded it - I've seen it happen in several Org's while a Major Incident Manager.

author: EEVblog
author_rank: Unknown
timestamp: December 15, 2016, 05:10:50 am
content: Quote from: djos on December 15, 2016, 04:49:08 amBingo.Everyone thinks that their hosting providers shit doesn't stink.

author: djos
author_rank: Unknown
timestamp: December 15, 2016, 05:15:24 am
content: Quote from: EEVblog on December 15, 2016, 05:10:50 amThe amusing thing about that was their systems total inability to deal with external power sags using UPS and start the GenSets up before the batteries drained - a total DC 101 Fail!

author: Muttley Snickers
author_rank: Unknown
timestamp: December 15, 2016, 05:20:27 am
content: I don't know enough about ISPs and hosting services to make any logical comment but I do know one thing, at least they had the decency to offer up an explanation and apology which is more than some other mobs do.On the day in question the error message I kept getting showed that the EEVblog Cloudfair Host located in Hong Kong was offline, I should have saved a snapshot to verify but didn't, I'm still getting the odd brief outage for a few minutes here and there as recently as this morning.

author: tautech
author_rank: Super Contributor
timestamp: December 15, 2016, 06:12:21 am
content: Quote from: Muttley Snickers on December 15, 2016, 05:20:27 amHello, mine showed Sydney, what's going on here ?Yep, I should have grabbed a screenshot too.

author: mtdoc
author_rank: Super Contributor
timestamp: December 15, 2016, 06:21:34 am
content: All I know is that of the 4 forums I regularly visit, this is the only one that has regular outages....

author: FrankBuss
author_rank: Unknown
timestamp: December 15, 2016, 08:24:57 am
content: Quote from: EEVblog on December 15, 2016, 05:10:50 amWell, that was a 2 hour outage at AWS because of a power outage. Hostgator was a 16 hour outage because of a configuration problem. Bigger datacenters like 1and1 have UPS, backup UPS and diesel generators for longer power outages.

author: djos
author_rank: Unknown
timestamp: December 15, 2016, 08:34:26 am
content: Quote from: FrankBuss on December 15, 2016, 08:24:57 amThe outage was caused by poor configuration of the DC BMS and UPS integration.The BMS didn't know how to handle a grid voltage sag (aka brown out) and just ran the ups batteries flat because it didn't know to call up the generators.I managed a tier 3 spec Colo DC up till a few years ago and this sort of Charlie Foxtrot just boggles the mind.

author: Ice-Tea
author_rank: Super Contributor
timestamp: December 15, 2016, 08:55:00 am
content: Just wondering, can't get to the "show new replies" page for a few days now... I get this:Code:502 - Bad GatewayWe are sorry for the inconvenience that this error may be causing you. We are aware of the issue and are working to resolve it, please be patient.There is no need to report this error.Thank you for your patience,Dave & gnifIs that part of this issue or something else?

author: MK14
author_rank: Super Contributor
timestamp: December 15, 2016, 08:56:31 am
content: Quote from: Ice-Tea on December 15, 2016, 08:55:00 amI had the same problem and solved it (today), by clicking on the web browsers refresh button.https://www.eevblog.com/forum/chat/show-new-reply-to-your-posts-causes-502-forum-error/msg1091765/?topicseen#msg1091765

author: sleemanj
author_rank: Super Contributor
timestamp: December 15, 2016, 09:04:39 am
content: Quote from: djos on December 15, 2016, 04:49:08 amYes, but with AWS you have a lot of power at your direct command and you can structure your systems to assume the sort of risk you want, from "availability zone in this region goes down, site is dead until it's fixed" to "this thing will keep running unless every Amazon region around the world has died".But of course, you have to be able to do that setup, if it's not your day job, better to leave somebody else to do it for you, if nothing else that there's a lot of assholes and botnets out there who will stop at nothing to try and kill your site/server/network. Some are smarter than others.Not all of them are as kind as one I caught today who kindly announced it's UserAgent as "WebFuck V2.1 T0PHackTeamwww.t0p.xyz" as it searched through some sites for exploits.

author: djos
author_rank: Unknown
timestamp: December 15, 2016, 09:07:13 am
content: Quote from: sleemanj on December 15, 2016, 09:04:39 amYou clearly didn't click on the link, they lost the whole Sydney D.C. Due to a poorly configured BMS not handling a grid voltage sag.Morale of the story, it can happen to anyone.

author: sleemanj
author_rank: Super Contributor
timestamp: December 15, 2016, 09:22:41 am
content: Quote from: djos on December 15, 2016, 09:07:13 amMy point was that the people who were using the Sydney DC could have used the AWS services at their disposal to prepare their system to be resilient to such failure of an entire region if they wanted to invest that time effort and expense.

author: djos
author_rank: Unknown
timestamp: December 15, 2016, 10:05:49 am
content: Quote from: sleemanj on December 15, 2016, 09:22:41 amGeographic diversity is no guarantee of 99.9999% uptime, I've managed 2 salesforce outages in 6 months caused by their Japanese D.C. Suffering networking failures.