Forum Posting Problem - Page 1
Forum section: electronics / general

author: EEVblog
author_rank: Unknown
timestamp: October 18, 2014, 03:18:57 am
content: Seems the database got corrupted and wouldn't allow posting. I think it's repaired now.Testing...

author: Skimask
author_rank: Super Contributor
timestamp: October 18, 2014, 03:27:02 am
content: Looks good...(P.S. Pull up your pants! Or spackle that thing shut!)

author: EEVblog
author_rank: Unknown
timestamp: October 18, 2014, 01:58:58 pm
content: Just check'n again

author: GEuser
author_rank: Frequent Contributor
timestamp: October 18, 2014, 03:20:46 pm
content: Up late too Dave, I thought that was just the innernet as I also had slow and disappearing (404's) at that time too .But just got this after clicking the post buttonConnection ProblemsSorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

author: EEVblog
author_rank: Unknown
timestamp: October 18, 2014, 10:39:29 pm
content: Quote from: GEuser on October 18, 2014, 03:20:46 pmThat one, apparently for the last for the last 3 hours was due to the MySQL server being locked up. A database service reboot fixed it.

author: EEVblog
author_rank: Unknown
timestamp: October 18, 2014, 11:23:53 pm
content: Hmm, all sorts of problems at the moment it seems. Had to reboot the MySQL server again, and there is a big load on the server. Something isn't right...

author: Pack34
author_rank: Frequent Contributor
timestamp: October 18, 2014, 11:29:59 pm
content: Quote from: EEVblog on October 18, 2014, 11:23:53 pmI remember having a similar issue with my company's web host. Maybe the physical server is starting to go. Any way to request a migration to a different machine?

author: EEVblog
author_rank: Unknown
timestamp: October 19, 2014, 12:11:19 am
content: Quote from: Pack34 on October 18, 2014, 11:29:59 pmI've been thinking about that. No real evidence that's the case, but it can't hurt I guess, but does seem to have had more downtime/crashes/reboots than I'd expect. I can ask...

author: Pack34
author_rank: Frequent Contributor
timestamp: October 19, 2014, 06:17:25 am
content: Quote from: wilfred on October 19, 2014, 12:28:53 amDo you think it could be something dns related? Intermittently pointing towards the website's old location? I don't know if an intermittent DNS problem is possible, just throwing out ideas.

author: atferrari
author_rank: Frequent Contributor
timestamp: October 19, 2014, 11:03:47 am
content: More than posting, my problem just to access the site. I cannot recall but there was a three-letters acronym involved, something like MPS or similar and an error code, starting with 5, but not sure really.

author: GEuser
author_rank: Frequent Contributor
timestamp: October 19, 2014, 11:07:58 am
content: I did not have to log on this evening as I could not log off early this morning as that message kept popping up .The site behaves real snappy here atm , instantaneous practically ..

author: T3sl4co1l
author_rank: Super Contributor
timestamp: October 19, 2014, 12:34:47 pm
content: Had this happen a few times, somewhat randomly. Seems to come and go.Tim

author: Neilm
author_rank: Super Contributor
timestamp: October 19, 2014, 07:16:57 pm
content: I have had 2 or 3 issues getting onto the EEVblog main site and the forum over the last few weeks.

author: tautech
author_rank: Super Contributor
timestamp: October 20, 2014, 09:57:40 pm
content: Quote from: Neilm on October 19, 2014, 07:16:57 pmMe too. Seems reliability is an issue ATM.Had this again this morning, though it seems all good now.Connection ProblemsSorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

author: atferrari
author_rank: Frequent Contributor
timestamp: October 21, 2014, 01:59:21 am
content: Quote from: tautech on October 20, 2014, 09:57:40 pmThe same here, several times today.

author: retiredcaps
author_rank: Super Contributor
timestamp: October 21, 2014, 02:02:51 am
content: I also had problems Monday Oct 20 starting at 10AM and ongoing to 10:30AM MST. I tried later in the afternoon around 2PM MST and was able to get on.I was able to pingwww.eevblog.com, but the forum and main page were down with the same error messages posted by others.

author: EEVblog
author_rank: Unknown
timestamp: October 21, 2014, 02:24:55 am
content: Yep, the server is having real problem at the moment, I need to speak with HostGator.The database issues I'm pretty sure are caused by the server going down, and when it does, there is always a chance a database table can get corrupted.

author: gnif
author_rank: Unknown
timestamp: October 21, 2014, 11:27:21 am
content: Hi All,I am sorry I have not been around to reply to any of this, I have been put out of action as I have been recovering from the whooping cough. I have suggested that Dave have all the RAM in the server replaced, the constant table crashes and now these major crashes point the finger at either RAM or HDD, but since the HDD is a RAID array and as far as I am aware is not reporting any issues, I would not expect this kind of crash (RAM is the easier to replace first too).

author: dr.diesel
author_rank: Super Contributor
timestamp: October 21, 2014, 12:05:33 pm
content: Quote from: wilfred on October 21, 2014, 12:01:35 pmIIRC it's a Linux box, single bit ECC errors would show up in dmesg.

author: EEVblog
author_rank: Unknown
timestamp: October 21, 2014, 12:08:42 pm
content: Hostgator are saying that it's the MySQL that is going down, and not the whole server. They are working on it...

author: gnif
author_rank: Unknown
timestamp: October 21, 2014, 01:18:03 pm
content: With regards to RAM, It looks like the server may be overcommited, I have reduced the total ram that MySQL is allowed to use, but this does not explain why it is getting blown way out. It seems to occur when there is heavy load on the server, I am seeing oom errors that is killing MySQL and hundreds of PHP processes running at the same time. I am investigating what PHP is doing during these periods.

author: bookaboo
author_rank: Frequent Contributor
timestamp: October 21, 2014, 01:21:17 pm
content: May be unrelated but the entire site was intermittent for me today from about 11.00am until 2.00PM GMT.

author: GEuser
author_rank: Frequent Contributor
timestamp: October 21, 2014, 01:24:11 pm
content: Doing some internal search's like 10 minutes ago a cupla times it came back > Database error/corruption retry later or something ..

author: gnif
author_rank: Unknown
timestamp: October 21, 2014, 01:34:18 pm
content: Quote from: bookaboo on October 21, 2014, 01:21:17 pmIt was related, logs show the database had crashed at that point.Quote from: GEuser on October 21, 2014, 01:24:11 pmThis may have occured while I was restarting the MySQL service to apply the memory changes.

author: T3sl4co1l
author_rank: Super Contributor
timestamp: October 21, 2014, 01:42:16 pm
content: Saw this earlier,Code:Table './eevblog_smp01/smf_members' is marked as crashed and should be repairedHope it's not hackers or something

author: GEuser
author_rank: Frequent Contributor
timestamp: October 21, 2014, 01:44:21 pm
content: Quote from: T3sl4co1l on October 21, 2014, 01:42:16 pmThat was another that popped up too

author: gnif
author_rank: Unknown
timestamp: October 21, 2014, 01:50:42 pm
content: Quote from: GEuser on October 21, 2014, 01:44:21 pmThanks for that, I will correct it now, that is a problem from the prior crash.Edit: This has been done, should not present again.Edit 2:I noted that even though we setup memcached way back, for some reason it is not in use. I have enabled it in SMF and have already noted quite a substantial reduction in server I/O. The forum also seems to be quite a bit faster for me, but I have not done any testing to properly confirm this.

author: EEVblog
author_rank: Unknown
timestamp: October 21, 2014, 01:58:48 pm
content: Quote from: T3sl4co1l on October 21, 2014, 01:42:16 pmThere are two issues at the moment, that have been persistent over the last two days.The first and most common is that error message listed above. I believe this is caused because the MySQL database shat itself for some reason, and when that happens there is a chance that any table that was being accessed will corrupt. Hence why you see different error message here. This requires a simple but manual table repair to fix. This seems to be happening every few hours at present.The second issue is that of a complete MySQL lockup. In this case the MySQL server has to be restarted. This will bring down not only the forum, but the wordpress blog too.I've disabled Cloudflare until all this gets fixed.

author: EEVblog
author_rank: Unknown
timestamp: October 21, 2014, 02:01:56 pm
content: Quote from: gnif on October 21, 2014, 01:50:42 pmI haven't fiddled with that. Although I did just update to SMF 2.0.9 yesterday.

author: gnif
author_rank: Unknown
timestamp: October 21, 2014, 02:10:23 pm
content: Quote from: EEVblog on October 21, 2014, 01:58:48 pmCloudflare has nothing in common with this issue, but with regards to the other issues people have had with it, I do not thing that it is a bad idea to disable it anyway.

author: EEVblog
author_rank: Unknown
timestamp: October 21, 2014, 02:29:19 pm
content: Quote from: gnif on October 21, 2014, 02:10:23 pmI've had constant problems with it myself.

author: gnif
author_rank: Unknown
timestamp: October 21, 2014, 02:45:04 pm
content: Leave it off then, I will keep a close eye on the server and see how things perform.

author: alimirjamali
author_rank: Regular Contributor
timestamp: October 21, 2014, 04:25:42 pm
content: I would agree with gnif that CloudFlare should have nothing to do with MySql issue. Currently CloudFlare cashing is disabled and only the DNS queries are resolved by them. Let's see if would help at all (which I doubt). I presume that you check mysql logs (/var/log/mysql/error.log || /var/log/mysqld.log || etc) for errors?Beside gnif, do you have sudoers (or any other support person) in other TimeZones? Maybe someone in UK or US you trust in person. Someone that could restart the bloody MySql service if needed!The value of posts on the Forum is substantial. Every time Forum crashes, I panic. Would you confirm that MySql databases and httpd data are backed up regularly (CRON, rsync, etc.)?Quote from: gnif on October 21, 2014, 01:18:03 pmIf it is not a huge secret, what is the output offree -mafter a fresh restart? I am curious to know how much RAM is installed on the server.

author: tautech
author_rank: Super Contributor
timestamp: October 25, 2014, 01:35:39 am
content: FYI: Had more of this early this morning:504 - Gateway TimeoutWe are sorry for the inconvenience that this error may be causing you. We are aware of the issue and are working to resolve it, please be patient.There is no need to report this error.Thank you for your patience,Dave & gnif

author: EEVblog
author_rank: Unknown
timestamp: October 25, 2014, 02:00:19 am
content: Quote from: tautech on October 25, 2014, 01:35:39 amApparently the FTDI thing got picked up on Hacker News and I presume the server got overloaded.

author: zapta
author_rank: Super Contributor
timestamp: October 25, 2014, 03:29:41 am
content: Quote from: EEVblog on October 25, 2014, 02:00:19 amOr FTDI decided to nuke also these forums...

author: tautech
author_rank: Super Contributor
timestamp: October 26, 2014, 02:41:53 am
content: Quote from: tautech on October 25, 2014, 01:35:39 amMore again for a short while today.

author: nanofrog
author_rank: Super Contributor
timestamp: October 26, 2014, 04:11:20 am
content: Quote from: tautech on October 25, 2014, 01:35:39 amI'm still getting this.

author: gnif
author_rank: Unknown
timestamp: October 26, 2014, 07:34:35 am
content: Quote from: wilfred on October 26, 2014, 03:37:41 amYou are sort of correct, that error about us knowing about it was when we had a prolonged window where we expected this to occur (server changes) and the error page was never fixed/updated when we were done.I have been monitoring the server closely to try to determine where the performance bottlekneck is in the current configuration, we initially assumed these issues were due to the Slashdot traffic, but it is becoming aparrent that there is something more going on.

author: tautech
author_rank: Super Contributor
timestamp: October 26, 2014, 07:40:38 am
content: QuoteYep for the last few weeks, sometimes for some hours.Today....infrequent short outages.

author: gnif
author_rank: Unknown
timestamp: October 26, 2014, 07:45:27 am
content: Quote from: tautech on October 26, 2014, 07:40:38 amI have just changed the website to store sessions in memcached instead of in the database on disk, this seems to be the bulk of I/O occuring and should make a noticable improvement to performance. If you get timeouts again please don't hesitate to PM me as I will notice that first. The next stage might just be a simple matter of increasing the number of apache processes as the gateway timeout occurs when Nginx can not communicate with the Apache backend (better solution yet would be to remove apache from the equasion, but would prefer not to as this starts to get outside of what cPanel will support).

author: T3sl4co1l
author_rank: Super Contributor
timestamp: October 27, 2014, 01:29:35 pm
content: Been getting a "session timeout, please attempt to reply again" message lately. Yes, I write overly long posts, but still...Tim

author: Thor-Arne
author_rank: Unknown
timestamp: October 27, 2014, 02:23:39 pm
content: I sometimes get "session timeout" when marking a board as read, so it's not the long posts.

author: EEVblog
author_rank: Unknown
timestamp: October 27, 2014, 09:11:21 pm
content: It went down again a few hours ago.Even the director of marketing at HostGator emailed me out of the blue and said they noticed it, and asked if it was due to FTDIgateI guess he's a viewerSo I think it is simply a spike in traffic every time FTDIgate gets linked somewhere.gnif has put in place caching that seems to be working well, but I guess there is only so much you can do with a single dedicated server.

author: alimirjamali
author_rank: Regular Contributor
timestamp: October 27, 2014, 09:23:32 pm
content: Quote from: EEVblog on October 27, 2014, 09:11:21 pmA proper engineer does not think but measure. Do you monitor RAM usage, Disk IO, CPU load? There are many utilities in Penguin for both real-time and logged measurement of system metrics (free/top/htop/iftop/iotop/etc.). Maybe it is time for an upgrade?

author: gnif
author_rank: Unknown
timestamp: October 28, 2014, 02:49:22 am
content: Quote from: alimirjamali on October 27, 2014, 09:23:32 pmWe do indeed monitor, and nothing directly points to an issue which is why it has been hard to find out what is going on. Since we corrected the issue with the database crashing we are getting some more constant error reporting from the server that is helping to track the issue down. At this point it seems that there is something wrong with PHP as we are now seeing tons of php segfaults in dmesg, I am sill investigating as to the cause.Edit: The outage that just occured was intentional, we just upgraded php to resolve a known bug with custom error handlers in the version of PHP we were running, I will conitnue to monitor the server and see if this resolves the issue that has been occuring.

author: gdewitte
author_rank: Unknown
timestamp: October 28, 2014, 02:37:38 pm
content: I'm consistently getting a "504-Gateway timeout" when I try to "Show unread posts…" Anyone else having this problem, or could it be something hosed on my PC? All the other links to various posts seem to work just fine.

author: Owen
author_rank: Contributor
timestamp: October 28, 2014, 08:31:07 pm
content: Quote from: gdewitte on October 28, 2014, 02:37:38 pmI'm getting this 504-Gateway timeout when i'm trying to enter "www.eevblog.com/forum/testgear/".

author: EEVblog
author_rank: Unknown
timestamp: October 28, 2014, 08:39:03 pm
content: Quote from: gdewitte on October 28, 2014, 02:37:38 pmYep, I get that too, but only on my machine at the lab, and only on Firefox. Not other browsers or computers.https://www.eevblog.com/forum/index.php?action=unreadworks fine though and does the same thing, but thehttps://www.eevblog.com/forum/unread/link does not work.

author: alimirjamali
author_rank: Regular Contributor
timestamp: October 28, 2014, 09:02:35 pm
content: Quote from: EEVblog on October 28, 2014, 08:39:03 pmThis is a server side issue and has almost nothing to do with your browser (Firefox) or internet connection (@lab).The reason you were able to get it through index.php is you virtually bypassed browser cashing by an alternative URL and got a fresh copy. The fresh copy might work fine or be another 504-Gateway timeout page.Hey gnif: Maybe it would be better to addMETA HTTP-EQUIV="Pragma" CONTENT="no-cache"to head section on the default 504-Gateway Timeout HTML page. Maybe add ameta http-equiv="refresh" content="300"tag too.

author: gnif
author_rank: Unknown
timestamp: October 29, 2014, 12:16:13 pm
content: Quote from: alimirjamali on October 28, 2014, 09:02:35 pmYou are quite correct, this however has not been a priority to tune as we have been looking for the root cause of the issue. I will adjust this a little later tonight, in the meantime I am preparing to remove apache from the equation and handle requests directly with Nginx, this should both improve performance and assist in determining where any potential bottleknecks are.

author: gdewitte
author_rank: Unknown
timestamp: October 29, 2014, 01:31:18 pm
content: Quote from: gdewitte onYesterdayat 01:37:38 AM

author: gnif
author_rank: Unknown
timestamp: October 29, 2014, 01:33:39 pm
content: Quote from: gdewitte on October 29, 2014, 01:31:18 pmI am preparing to remove apache from the equation and handle requests directly with Nginx, this should both improve performance and assist in determining where any potential bottleknecks are.

author: alimirjamali
author_rank: Regular Contributor
timestamp: October 29, 2014, 05:11:15 pm
content: Quote from: gnif on October 29, 2014, 01:33:39 pmHey gnif. I see you have enabled CloudFlare cashing again. It is not just DNS anymore.So you are going to ditch Apache in favor of Nginx. I am happy that I am not in your position to translate all of those nasty .htaccess and Apache VirtualHost config files. You are going to test it on a local copy before applying them?right? We have some EEVBlog Forumoholics who will be pissed off if Forum is down more than 1 day.<ConspiracyTheory>Maybe Dave is DDoSed? Dave's WordPress Blog and SMF Forum are popular but not that much popular which Apache could not handle. There are less than 2000 Online Users+Guests on Forum. Considering that SMF User online time threshold is 15 minutes (by default) which implies that they do not fetch data simultaneously, Apache should be able to handle them easily.2000 users / 15 minutes / 60 seconds * Approx. 20 requests per users = Approx. 44 requests per second = piece of cake for Apache.Even before the #FTDIGate, Dave had made many foes. Maybe if you double check IP of users, you might see 90% from some FTDI server(What Da FTDI).</ConspiracyTheory>Good luck.

author: gnif
author_rank: Unknown
timestamp: October 29, 2014, 05:19:41 pm
content: Quote from: alimirjamali on October 29, 2014, 05:11:15 pmThis was Dave's decision to do so, it has nothing to do with the gateway issues.Quote from: alimirjamali on October 29, 2014, 05:11:15 pmAlready translated, and live, see the new thread.Quote from: alimirjamali on October 29, 2014, 05:11:15 pmNo DDoS and we rate limit per IP anyway, so they would need to have way more servers then you would expect to saturate the server. Since this is a cPanel server it is tuned to host multiple small to medium sized sites for mass reselling, not one large site. It builds Apache in such a way as to suit this purpose and in this instance it is a hinderance. Moving away from Apache to Nginx + PHP-FPM allows us to escape the bounds of the cPanel paradigm.

author: T3sl4co1l
author_rank: Super Contributor
timestamp: November 06, 2014, 08:48:41 am
content: Quote from: T3sl4co1l on October 27, 2014, 01:29:35 pmIt's back

author: gnif
author_rank: Unknown
timestamp: November 06, 2014, 04:47:57 pm
content: Quote from: T3sl4co1l on November 06, 2014, 08:48:41 amI will adjust how memcached is storing sessions and increase the timeout, this should correct this.

author: T3sl4co1l
author_rank: Super Contributor
timestamp: November 07, 2014, 01:39:30 am
content: Thanks!

author: EvilGeniusSkis
author_rank: Regular Contributor
timestamp: November 13, 2014, 01:36:43 am
content: could someone have been using something more sophisticated than a simple DDoS, such as SlowLoris or RUDY?

author: gnif
author_rank: Unknown
timestamp: November 13, 2014, 05:02:47 am
content: Quote from: EvilGeniusSkis on November 13, 2014, 01:36:43 amCertainly not, logs clearly indicated that Apache was running out of connections well before server load was an issue, Apache is not exposed to the internet as it was behind Nginx which was the reverse proxy. If it was a DoS attack such as SlowLoris or RUDY Nginx would also have stopped responding.Put simply, Apache is software that should be put down, it was good in it's day, but it is well and truly past it's expiration date. Please do not think that I believe this due to a limited understanding of Apache, I am extremely well versed with it, and I am the maintainer/developer ofmod_rpafwhich is in use by hundreds if not thousands of large scale deployments around the globe, so I know Apache from a code perspective, not just a deployment perspective.

author: alimirjamali
author_rank: Regular Contributor
timestamp: November 13, 2014, 06:05:08 am
content: Quote from: EvilGeniusSkis on November 13, 2014, 01:36:43 amFTDI produces many chipsfor Androidsmart phones. They could have turned millions of Android phones to zombies and attacked Dave's forum in retaliation to the #FTDIgatep.s. I know that it is technically impossible; However, conspiracy theories are simply cool.

author: gnif
author_rank: Unknown
timestamp: November 13, 2014, 06:54:34 am
content: Quote from: alimirjamali on November 13, 2014, 06:05:08 amHaha, you literally made me LOL.In all this was a good exercise as the server is now running better then ever for pretty much everyone, even those of us who are over the pond and have to already deal with large latency due to the transmission delays.

author: EvilGeniusSkis
author_rank: Regular Contributor
timestamp: November 14, 2014, 01:39:55 am
content: this whole thing reminds me of something my multimedia teacher would sayQuotethe reason I say this is, to me anyway, it seem that the problem was fixed without anyone figuring out what the problem was exactly. am i right?also is there a forum features wishlist thread anywhere?

author: gnif
author_rank: Unknown
timestamp: November 14, 2014, 11:33:12 am
content: Quote from: EvilGeniusSkis on November 14, 2014, 01:39:55 amI have very clearly stated what the problem was, Apache simply is not capable of serving the sheer number of PHP requests the website has as it is archaic in its design and can not take advantage of modern hardware correctly.

author: atferrari
author_rank: Frequent Contributor
timestamp: November 15, 2014, 01:23:28 pm
content: Hola gnifI want to stop receiving notifications of new replies to some threads.I clicked on "UNNOTIFY" at the top of them but I still receive notifications.One of the threads is this very one.Gracias for any help.