504 Gateway Time-out - Page 1
Forum section: electronics / general

author: AlphZeta
author_rank: Regular Contributor
timestamp: May 03, 2013, 12:49:36 pm
content: I have been getting the "504 Gateway Time-out" error quite a bit during the past week or so. A couple of days ago, the site was inaccessible for a couple of hours in the morning.Any one else have this issue?

author: c4757p
author_rank: Super Contributor
timestamp: May 03, 2013, 12:54:02 pm
content: Yes

author: PeterG
author_rank: Frequent Contributor
timestamp: May 03, 2013, 12:54:44 pm
content: Same happened here.

author: ecat
author_rank: Frequent Contributor
timestamp: May 03, 2013, 12:57:43 pm
content: Yep, 2nd time this week.

author: enz
author_rank: Regular Contributor
timestamp: May 03, 2013, 01:11:22 pm
content: Yes, same here.Last one was just 15 minutes ago.Took a couple of minutes until it worked again.Martin

author: ddavidebor
author_rank: Super Contributor
timestamp: May 03, 2013, 01:11:35 pm
content: Ihere too

author: c4757p
author_rank: Super Contributor
timestamp: May 03, 2013, 01:15:10 pm
content: It seems Dave is aware of the problem.

author: Hydrawerk
author_rank: Super Contributor
timestamp: May 03, 2013, 01:49:13 pm
content: It happened to me a few times in last week.

author: EEVblog
author_rank: Unknown
timestamp: May 03, 2013, 01:55:37 pm
content: Geoff (head penguin) is aware of the issue.The last major error was due to a "stuck backup process" apparently.I didn't know it has happened multiple times.So only in the last week or so?

author: AlphZeta
author_rank: Regular Contributor
timestamp: May 03, 2013, 02:09:24 pm
content: Quote from: EEVblog on May 03, 2013, 01:55:37 pmIt certainly seems that way. This error usually means a back-end setting issue though.

author: gnif
author_rank: Unknown
timestamp: May 04, 2013, 12:42:49 am
content: Hi guys,Sorry for this, I have applied a few changes that should help with this, but only time will tell if it actually resolves it. There is also a backup process that runs that could be causing it, but I am yet to perform further testing to see if it is occurring during the time that the backup runs.

author: BravoV
author_rank: Super Contributor
timestamp: May 05, 2013, 02:20:52 pm
content: I thought these days backup process is seamless ?I mean no matter what happened at that backup process even its failed or stuck-ed, it will not affect the main active operating apps right ?Remembered had to shutdown the active running program when backup process was initiated, but that was in DOS era.

author: EEVblog
author_rank: Unknown
timestamp: May 05, 2013, 02:26:06 pm
content: Quote from: BravoV on May 05, 2013, 02:20:52 pmIt never used to.I've been running the same cpanel backup process for maybe 9 months now.I can disable this backup for a while and see if goes away, or initiate one on demand and see it shuts the sever down again.

author: bingo600
author_rank: Super Contributor
timestamp: May 05, 2013, 06:55:04 pm
content: I was hit by 504 today , for the first time ....Consistent 504's for 5 min or more , so i did something else for a few hours.And now it's working/Bingo

author: metalphreak
author_rank: Frequent Contributor
timestamp: May 06, 2013, 01:16:24 pm
content: 502s and 504s for the last 20mins... if this helps pinpoint a time.

author: ecat
author_rank: Frequent Contributor
timestamp: May 06, 2013, 01:19:16 pm
content: Trouble here started at or before 13:25 UK time (BST)

author: AlphZeta
author_rank: Regular Contributor
timestamp: May 06, 2013, 01:19:48 pm
content: Yep, this morning I received the 504 error again since 8:20AM EST and the site was just accessible again about 10 minutes earlier (around 9:10AM).

author: EEVblog
author_rank: Unknown
timestamp: May 06, 2013, 01:24:04 pm
content: I just got the problem again now.I have now disabled siteautobackup.com for the next few days to see if this is the culprit.Please report any future errors here.

author: EEVblog
author_rank: Unknown
timestamp: May 08, 2013, 01:17:18 pm
content: Ok, it happened again tonight. At almost the exact same time as last night.external backups were disabled.HostGator could not access my server and rebooted it.Still no idea what the problem is.gnif?

author: firewalker
author_rank: Super Contributor
timestamp: May 08, 2013, 01:19:32 pm
content: It Keeps happening to me. Just happened again. 3~4 times per day. For about 3~5 minutes each time.Alexander.

author: EEVblog
author_rank: Unknown
timestamp: May 08, 2013, 01:27:29 pm
content: Quote from: hammy on May 08, 2013, 01:26:28 pmYes, exact same for me. The server reboot fixed it.

author: gnif
author_rank: Unknown
timestamp: May 08, 2013, 01:27:52 pm
content: Quote from: EEVblog on May 08, 2013, 01:17:18 pmI am seeing a stack of PHP processes running trying to serve requests, server out of RAM and very hard to get to respond. Restarting Apache fixes it, but I am still to identify the cause. I apologise for the issue but this one is a little hard to track down due to the symptoms when it occurs.

author: firewalker
author_rank: Super Contributor
timestamp: May 08, 2013, 01:30:44 pm
content: The 540, was due the reset of the server? Was the server alive for anyone?Alexander.

author: EEVblog
author_rank: Unknown
timestamp: May 08, 2013, 01:37:33 pm
content: Quote from: gnif on May 08, 2013, 01:27:52 pmWell, php is likely either the forum, or Wordpress, right?I wonder if I installed a rouge plugin in the last week or so, can't recall...

author: EEVblog
author_rank: Unknown
timestamp: May 08, 2013, 02:08:05 pm
content: Response from HostGator:Quote

author: HackedFridgeMagnet
author_rank: Super Contributor
timestamp: May 09, 2013, 01:22:20 pm
content: Happend again, at about 11pm AEST. 2013-05-09504 Gateway Time-outnginxI did a trace route from telstra to eevblog.com (198.1.64.143)1 gigabitethernet3-3.exi1.melbourne.telstra.net (203.50.77.49) 0.232 ms 0.275 ms 0.242 ms2 bundle-ether3.exi-core1.melbourne.telstra.net (203.50.80.1) 1.742 ms 2.045 ms 2.992 ms3 bundle-ether12.chw-core2.sydney.telstra.net (203.50.11.74) 15.234 ms 15.412 ms 15.986 ms4 bundle-ether1.oxf-gw2.sydney.telstra.net (203.50.6.90) 15.605 ms 15.412 ms 15.985 ms5 tengigabitethernetx-0.syd-core04.sydney.reach.com (203.50.13.18) 17.603 ms 19.412 ms 15.970 ms6 i-0-3-2-0.sydo-core02.bi.telstraglobal.net (202.84.220.189) 17.982 ms 19.412 ms 15.980 ms7 i-0-2-0-0.1wlt-core01.bx.telstraglobal.net (202.84.141.146) 158.024 ms 155.455 ms 156.030 ms8 i-0-4-0-0.eqla01.bi.telstraglobal.net (202.84.251.174) 160.400 ms 159.339 ms 159.139 ms9 cogent-peer.eqla01.pr.telstraglobal.net (134.159.63.198) 160.773 ms 160.704 ms 160.651 ms10 te0-1-0-1.mpd21.lax01.atlas.cogentco.com (154.54.0.217) 160.648 mste0-2-0-2.ccr21.lax01.atlas.cogentco.com (154.54.3.141) 160.704 mste0-2-0-2.mpd21.lax01.atlas.cogentco.com (154.54.0.213) 160.579 ms11 be2021.ccr21.lax04.atlas.cogentco.com (154.54.87.250) 161.077 msbe2019.ccr21.lax04.atlas.cogentco.com (154.54.88.10) 161.077 msbe2021.ccr21.lax04.atlas.cogentco.com (154.54.87.250) 161.329 ms12 acedc.net (38.104.83.18) 154.331 ms 154.458 ms 154.278 ms13 tg1-1.ar01.prov.acedc.net (199.58.196.37) 173.391 ms 171.946 ms 176.891 ms14 199.58.199.114 (199.58.199.114) 172.015 ms 171.954 ms 220.353 ms15 mlx01.prov.unifiedlayer.com (69.195.64.41) 172.013 ms 171.940 ms 171.770 ms16 *couldn't see the problem though so it must be at the hosting site.Nginx?

author: EEVblog
author_rank: Unknown
timestamp: May 09, 2013, 01:23:15 pm
content: Yep, right on cue again, 10:30pm sydney time.Let's hope HostGators monitoring software caught something this time.

author: Hydrawerk
author_rank: Super Contributor
timestamp: May 09, 2013, 01:28:01 pm
content: It happened again.

author: Noize
author_rank: Regular Contributor
timestamp: May 09, 2013, 01:43:08 pm
content: I'm getting 504's regularly at 1.30pm GMT every day including today. That ties in with 10.30pm sydney time.Quote from: EEVblog on May 09, 2013, 01:23:15 pm

author: KD0CAC John
author_rank: Frequent Contributor
timestamp: May 09, 2013, 02:15:48 pm
content: I've been having this the last few days , and tried this morning off & on for around 1/2 hr. trying using search , bookmarks , cleared cash .Then did other stuff and got on without an issue just now .This would have been about an hour ago , if that helps

author: HackedFridgeMagnet
author_rank: Super Contributor
timestamp: May 09, 2013, 02:20:05 pm
content: dont know if it's relevanthttp://apache.slashdot.org/story/13/05/09/003236/backdoor-targeting-apache-servers-spreads-to-nginx-lighttpd

author: firewalker
author_rank: Super Contributor
timestamp: May 10, 2013, 12:53:02 pm
content: Happened again today. It just recovered.Alexander.

author: GeoffS
author_rank: Unknown
timestamp: May 10, 2013, 12:54:42 pm
content: Just happened to me, right on 22:29 (local/Sydney time) and lasted at least 15 minutes.

author: KD0CAC John
author_rank: Frequent Contributor
timestamp: May 10, 2013, 01:02:59 pm
content: Came right up this morning

author: firewalker
author_rank: Super Contributor
timestamp: May 10, 2013, 02:18:19 pm
content: For about 30 minutes there was the 504 error and after that an SMF database connect error.Alexander.

author: EEVblog
author_rank: Unknown
timestamp: May 10, 2013, 02:20:39 pm
content: gnif think he found it.The server stats were being all updated at 10:30 each day and chewing up all the RAM.Now i recall turning on analog stats a little while back...Awstats and Webalizer have always been running from day 1.

author: firewalker
author_rank: Super Contributor
timestamp: May 10, 2013, 03:50:58 pm
content: It was out for about an hour. SMF database connection error. It was gnif trying to nail it?Alexander.

author: robrenz
author_rank: Super Contributor
timestamp: May 10, 2013, 03:55:07 pm
content: Realy bad here, on and mostly off for the last 2 hours

author: AlphZeta
author_rank: Regular Contributor
timestamp: May 10, 2013, 05:00:06 pm
content: Quote from: firewalker on May 10, 2013, 03:50:58 pmYeah, same here this morning. database connection error to the forum and the main site.

author: blewisjr
author_rank: Frequent Contributor
timestamp: May 10, 2013, 05:19:45 pm
content: Same here got wacked by some 504/505 and SF db errors earlier to day seem to have cleared up now.

author: gnif
author_rank: Unknown
timestamp: May 10, 2013, 05:46:06 pm
content: The SMF errors were because I had locked the database so I could convert some tables for performance tuning.

author: EEVblog
author_rank: Unknown
timestamp: May 10, 2013, 10:30:41 pm
content: OK, it'sshouldbe fixed now. Stats are disabled, so no more operation should run at 10:30pm.Yes, those database connection errors were expected during a reorg of the database.It seems reasonable to assume that me turning on analog stats a while back is what started the rot. Apparently I don't have enough RAM for that, or the stats are broken somehow.Please report any further errors here, there shouldn't be any again...Thanks to gnif yet again for staying up late and finding and fixing it all

author: EEVblog
author_rank: Unknown
timestamp: May 11, 2013, 11:21:27 pm
content: Did anyone have any issues last night?If not, I declare the problem fixed!

author: amspire
author_rank: Super Contributor
timestamp: May 11, 2013, 11:25:16 pm
content: I was online last night at the usual drop-out time and it worked perfectly.

author: c4757p
author_rank: Super Contributor
timestamp: May 11, 2013, 11:26:37 pm
content: Same here, for a loose definition of "night"

author: xrunner
author_rank: Super Contributor
timestamp: May 11, 2013, 11:27:51 pm
content: I got a 505 Gateway Time-out, but the 504 Gateway time-out seems to be fixed.Just kidding - no issues here.

author: EEVblog
author_rank: Unknown
timestamp: May 11, 2013, 11:29:25 pm
content: Oh, and BTW, it WAS my faultI turned on analog stats which is almost certainly the culprit.

author: metalphreak
author_rank: Frequent Contributor
timestamp: May 12, 2013, 02:50:02 pm
content: You should keep a small log somewhere of any changes you do to the server. Perhaps just a text file with:xx-04-2013 Installed analog statsMight help narrow down issues in the future

author: KD0CAC John
author_rank: Frequent Contributor
timestamp: May 12, 2013, 03:02:59 pm
content: Quote from: EEVblog on May 11, 2013, 11:29:25 pmBad ozzie , Bad ozzieNo problem for a few days not , OK good Dave , good Dave .

author: gnif
author_rank: Unknown
timestamp: May 13, 2013, 11:41:11 pm
content: Quote from: wilfred on May 11, 2013, 12:57:40 amLinux does log a lot on a stock system, but cPanel is one of those bastardised systems where all the packages are customised and compiled from source, so what logging there is, it is all over the shop.Quote from: wilfred on May 11, 2013, 12:57:40 amI work a late shift anyway, so it is no big deal for me. Dave is using Google analytics to collect his stats, which IMO does a much better job then the cPanel offerings while offloading all the stats generation to their servers, so I have simply disabled all stats generation which will prevent this from occurring again.

author: xrunner
author_rank: Super Contributor
timestamp: May 20, 2013, 12:31:46 am
content: Error messages seen this PM -504 Gateway Time-outConnection ProblemsSorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

author: gnif
author_rank: Unknown
timestamp: May 20, 2013, 12:35:27 am
content: Quote from: xrunner on May 20, 2013, 12:31:46 amThanks, I will see if I can locate the issue. Might just need to raise the number of allowed db connections.

author: gnif
author_rank: Unknown
timestamp: May 20, 2013, 12:42:22 am
content: The max connections had indeed been exceeded, I have increased it from 150 to 300, this should prevent this from occurring again.

author: darrylp
author_rank: Regular Contributor
timestamp: May 20, 2013, 01:03:35 am
content: Just had SMF rejection ( about 20 mins ago )--Darryl

author: gnif
author_rank: Unknown
timestamp: May 20, 2013, 01:07:19 am
content: Quote from: darrylp on May 20, 2013, 01:03:35 amThat was because to increase the value I had to restart MySQL. If you get any more as of my last post, please let me know.

author: justanothercanuck
author_rank: Frequent Contributor
timestamp: May 20, 2013, 02:30:13 am
content: Quote from: gnif on May 20, 2013, 12:42:22 amIs the forum still running with persistent connections enabled? Also, 150 is the linux mysql default. Sounds a little low to me.

author: EEVblog
author_rank: Unknown
timestamp: May 20, 2013, 03:40:07 am
content: Quote from: justanothercanuck on May 20, 2013, 02:30:13 amYes, "Use a persistent connection" is enabled in SMF.

author: gnif
author_rank: Unknown
timestamp: May 20, 2013, 04:04:44 am
content: Quote from: justanothercanuck on May 20, 2013, 02:30:13 am150 is not low for a server dedicated to a single site, this limit is imposed simply to prevent abuse of MySQL, raising it when you don't need to is just a bad idea as if your server starts to come under attack you have no indication as to the cause. Best practice is to use the default 150, if it is exceeding it's limit then find out why, if it is legitimate then increase the limit, otherwise fix the problem.

author: GeoffS
author_rank: Unknown
timestamp: May 22, 2013, 06:41:20 am
content: Just had another 504 error.Worked OK on refresh but a little slow right now.

author: gnif
author_rank: Unknown
timestamp: May 22, 2013, 06:49:10 am
content: Quote from: GeoffS on May 22, 2013, 06:41:20 amThanks for the heads up, this time it is due to the server performing it's daily backup. Perhaps Dave should look at an alternative backup solution such as R1Soft which does an incremental backup.

author: gnif
author_rank: Unknown
timestamp: May 22, 2013, 08:53:14 am
content: Quote from: wilfred on May 22, 2013, 08:37:25 amThe error resolved was not a 504, it was the SMF database connection error that PHP was reporting via Apache, but to answer your question...504 = Gateway timeout, which can occur for many many reasons, all of which result in the front end reverse proxy (Nginx) from being unable to get a response out of Apache within a decent time frame.When cPanel backs up the user accounts (only one in this instance), it makes a tar file of the entire home directory for that user, and then compresses it with gzip. The eev account is multiple GB in size, and consists of thousands of files, try and compress that kind of data on your desktop and you will understand why it causes a load issue.When cPanel does this, it tries to prevent the load from exceeding sane levels, but it uses the 5 minute average to do this, and instead of slowing down the compression with an artificial sleep in the loop, it just halts it until that 5 minute average drops below the sane level again, which means that each time it resumes, it spikes back up to 100% CPU load.Since the site is handling so many visitors, someone is bound to get these time-outs during the backup process and it will be completely random who sees it. There are a lot of factors involved here, but the cause of these rare, random 504's are due to the backup process hogging all the CPU and I/O.Edit: Also, we did have reports of the 504 errors in the past. When there are other more major issues at hand one does not notice the 1 in a thousand error amongst the other thousand errors for the other issue.

author: gnif
author_rank: Unknown
timestamp: May 22, 2013, 09:48:20 am
content: Quote from: wilfred on May 22, 2013, 09:26:04 amUnfortunately it is not that simple, cPanel already runs it at a lower nice level and if it ran as another user it would not have enough access to backup the user's account/files as many of the files are not owned by the user.The nice level just tells the scheduler how important the process is that it gets CPU time, not how much time it gets. So if server load is low, and the backup process is assigned a chunk of CPU time, and a stack of requests come in at the same time, it is too late, the time slice has been allocated to the backup process and one of those requests is going to suffer.There are only two solutions to this... throw more hardware at the server in the form of CPUs, or look at a better backup solution.

author: EEVblog
author_rank: Unknown
timestamp: May 22, 2013, 11:01:18 am
content: Quote from: gnif on May 22, 2013, 06:49:10 amYes, I just re-enabled siteautobackup.com today.The thing is I've been using that for almost a year now, presumably without issue?

author: EEVblog
author_rank: Unknown
timestamp: May 22, 2013, 11:07:33 am
content: Quote from: gnif on May 22, 2013, 09:48:20 amThe server is currently an Intel Xeon E3-1220LV2 2.3GHz with 4GB RAM.An Intel Xeon E3-1265L 2.5GHz with 8GB is an extra $95/month.So presumably that money would buy me some other schmick external backup solution?

author: gnif
author_rank: Unknown
timestamp: May 22, 2013, 11:34:59 am
content: Quote from: EEVblog on May 22, 2013, 11:07:33 amLet me get back to you on that, I have been developing a potential solution that might be suitable.

author: gnif
author_rank: Unknown
timestamp: May 22, 2013, 12:14:32 pm
content: Quote from: wilfred on May 22, 2013, 12:03:01 pmThis is not the issue, cPanel already re-nices the process to low priority, and it is not I/O bound. I agree that incremental is not the best solution, and that cPanel is not very good at it anyway (read: I don't trust it from bad experience).Quote from: wilfred on May 22, 2013, 12:03:01 pmAgreed, which is why I am looking at a better solution that tracks block level changes like R1Soft does.Quote from: wilfred on May 22, 2013, 12:03:01 pmYou would be correct, once Linux hits 100% load or higher, it snowballs to the point where you basically have to reboot the machine.

author: EEVblog
author_rank: Unknown
timestamp: May 22, 2013, 12:42:55 pm
content: Quote from: gnif on May 22, 2013, 11:34:59 amBare in mind that I really like the cpanel backup, as it means that I can potentially restore everything myself fairly easily if it came to that. I can either restore via cpanel in one click, or optionally access the .sql database files manually etcIf an alternative solution has a high penguin requirement, then that means I'm reliant upon someone else if shit hits the fan.

author: EEVblog
author_rank: Unknown
timestamp: May 22, 2013, 12:45:26 pm
content: Quote from: wilfred on May 22, 2013, 12:03:01 pmI'm not sure I quite understand the load issue. I've been running cpanel backups from day dot, both on cheap $10 shared servers, cloud servers, and now an expensive dedicated server, and have never had any reported issues. Even when something like the shared server is running at almost 100% CPU usage 24/7.

author: gnif
author_rank: Unknown
timestamp: May 23, 2013, 04:55:31 am
content: Quote from: EEVblog on May 22, 2013, 12:45:26 pmThis is because on a shared cloud server, the backup process is normally shared across a server or multiple servers that are many times more powerful then your single dedicated. Normally there is only 1-2% of users on shared systems that use any decent amount of traffic, or cause a decent amount of load, so when your backup runs on those powerful servers they run without a hitch because the server has more resources then it actually requires for the general hosting it is performing.I used to work for a company where we had a shared hosting environment that consisted of 400+ users on each single server, if we ever had a user that started to cause load issues, or performance issues we would sell them into a dedicated server to prevent them from causing problems for the other clients. We also had a 2nd server that ran as a MySQL slave to each primary server which was used to take the MySQL backup from, as while cPanel makes this backup tables in the database get locked which prevent the website from being able to query them. Eventually we moved to R1Soft because it resolved every issue we had with backup performance.As for a solution that even you can use, R1Soft is one, it integrates into cPanel and you may even find that HostGator offer it as an addon service, it is truly superior to cPanel's tar & gzip solution and much easier to restore individual files from. If you go down this route I also recommend you spend the few $ extra to get the MySQL addon which also allows you to restore individual tables, and ensures you get a consistent snapshot of the database in the backup.Edit: Also, with R1Soft, you can restore the entire server from the backup, not just the user accounts.

author: GeoffS
author_rank: Unknown
timestamp: May 24, 2013, 07:02:49 am
content: Another 504 incident to add to the list.Occurred at 16:26 Sydney time.

author: BravoV
author_rank: Super Contributor
timestamp: May 24, 2013, 07:04:28 am
content: Quote from: GeoffS on May 24, 2013, 07:02:49 amYep, same here.

author: GeoffS
author_rank: Unknown
timestamp: May 25, 2013, 07:45:21 am
content: And yet another that lasted from 16:45 (or so) . Lasted quite a while

author: notsob
author_rank: Frequent Contributor
timestamp: May 25, 2013, 08:54:49 am
content: Same here, seems to be getting worse over the last few days.

author: Lightages
author_rank: Unknown
timestamp: May 25, 2013, 08:24:11 pm
content: I have been seeing them regularly now too and a couple have happened after I have composed a reply to a post and then lost the post. It is getting very annoying. The last one I saw lasted 30 minutes more or less.

author: jancumps
author_rank: Unknown
timestamp: May 25, 2013, 08:55:01 pm
content: same here 15 hours ago

author: retiredcaps
author_rank: Super Contributor
timestamp: May 25, 2013, 10:15:04 pm
content: Quote from: Lightages on May 25, 2013, 08:24:11 pmI had a 504 timeout as well at the same time everyone else is reporting.As for losing your post, I always compose long responses in notepad and save regularly. Then I paste.

author: EEVblog
author_rank: Unknown
timestamp: May 25, 2013, 10:46:33 pm
content: Ok, this is getting ridiculous. I will now permanently disable autositebackup.com and will have to find another backup solution.The site will no longer be backed up daily until I can find something else.

author: lemmegraphdat
author_rank: Frequent Contributor
timestamp: May 26, 2013, 12:11:06 am
content: Quote from: EEVblog on May 25, 2013, 10:46:33 pmNo worries Dave. You'll fix it. A one man band has a lot on his hands.

author: EEVblog
author_rank: Unknown
timestamp: May 26, 2013, 12:34:48 am
content: The R1soft Cpanel plugin doesn't sound good:http://www.webhostingtalk.com/showthread.php?t=1090512

author: EEVblog
author_rank: Unknown
timestamp: May 26, 2013, 12:41:16 am
content: WHM has to option to automatically backup the server. I'm not sure in what format it does this though. It supports both local (pointless?) or remote FTP backup.http://docs.cpanel.net/twiki/bin/view/11_32/WHMDocs/ConfigBackupIf I had a remote FTP directory on another server somewhere, then that would do the job. But unless it could somehow only keep the last X backups, the backup drive space would quickly fill up.I have other servers I could presumably FTP into, but they are just cheap shared servers, and don't allow storage of backups as part of their T&C.

author: Monkeh
author_rank: Super Contributor
timestamp: May 26, 2013, 12:43:50 am
content: Why not send the backups home?And please please don't use FTP. Please.

author: EEVblog
author_rank: Unknown
timestamp: May 26, 2013, 01:04:46 am
content: Quote from: Monkeh on May 26, 2013, 12:43:50 amHow do I do that?I don't run an FTP server 24/7, or at all.Also, a 7GB+ backup each day will kill my entire monthly download budget.QuoteWHM backup only supports FTP.

author: Monkeh
author_rank: Super Contributor
timestamp: May 26, 2013, 01:07:56 am
content: Quote from: EEVblog on May 26, 2013, 01:04:46 amWell, you need to run a service to receive it. Or do the backup by having one of your machines connect to the server.QuoteAbandon ship. DO NOT use FTP. I don't care what the software supports, FTP leaves your arse hanging in the breeze. It's the most insecure thing you can use.

author: EEVblog
author_rank: Unknown
timestamp: May 26, 2013, 01:14:07 am
content: Ah, siteautobackup.com also supports WHM backup as well. Perhaps that will work better, as presumably it would be using the WHM backup facility instead of cPanel backup. Will requite more disk space though as I presume it backs up the entire server instead of just the cPanel account?

author: EEVblog
author_rank: Unknown
timestamp: May 26, 2013, 01:15:42 am
content: Quote from: wilfred on May 26, 2013, 01:09:17 amI have no choice. I cannot have my entire server go down every day due to this backup process.

author: EEVblog
author_rank: Unknown
timestamp: May 26, 2013, 01:17:15 am
content: Quote from: Monkeh on May 26, 2013, 01:07:56 amSo a lot of people keep telling me.I don't think the world is going to end if I get WHM to connect toanotherserver via FTP and send the backup there.If you have a better option for doing my backups then please mention it.

author: EEVblog
author_rank: Unknown
timestamp: May 26, 2013, 01:26:26 am
content: I have enabled autositebackup.com to do a daily back of my WHM account, and existign cpanel account back has been disabled again. I'll see how much space it takes and if it brings down the server like cpanel backup does.

author: justanothercanuck
author_rank: Frequent Contributor
timestamp: May 26, 2013, 01:29:45 am
content: Can't you just script it? Do a mysqldump for the database, and tar up the webroot... the rest should just be the OS, which is inconsequential for a website so long as they provide apache with php (and maybe cgi) and mysql.

author: Monkeh
author_rank: Super Contributor
timestamp: May 26, 2013, 02:33:31 am
content: Quote from: EEVblog on May 26, 2013, 01:17:15 amYou've had one security failure recently, FTP is begging for more. Time to double check I used a unique password here..QuoteI don't know exactly what you need to backup or to where, and frankly I can't think of any simple one button solutions worth using. The already mentioned MySQL replication idea should work nicely for the databases, though.

author: EEVblog
author_rank: Unknown
timestamp: May 26, 2013, 02:37:31 am
content: Quote from: justanothercanuck on May 26, 2013, 01:29:45 amMaybe, if I learn how to script.But the issues remains, where does the backup get sent and how?Backing up to the servers own hard drive is of course pointless.I also cannot download to my own machine due to monthly bandwidth limitations.The WHM backup seems to be the best option at the moment, lets see if it works.

author: Monkeh
author_rank: Super Contributor
timestamp: May 26, 2013, 02:39:18 am
content: Quote from: EEVblog on May 26, 2013, 02:37:31 amYour various sites have that much data moving through that a daily delta is a significant amount of bandwidth?

author: EEVblog
author_rank: Unknown
timestamp: May 26, 2013, 02:39:49 am
content: I just manually started the WHM backup via siteautobackup.com, lets see if the server holds up.I have no idea how long it will take.

author: EEVblog
author_rank: Unknown
timestamp: May 26, 2013, 02:42:53 am
content: Quote from: Monkeh on May 26, 2013, 02:39:18 amNo, but please tell me how to do a daily delta...Remember, I am not an IT geek, have no desire to learn, no desire to leave my machine on 24/7, and no desire to manually do anything. I just want my website and databases automatically backed up daily, somewhere off site to my server.The WHM backup does not support delta to a remote host via FTP.

author: Monkeh
author_rank: Super Contributor
timestamp: May 26, 2013, 02:49:10 am
content: Quote from: EEVblog on May 26, 2013, 02:42:53 amA tool such as rdiff-backup run on a daily basis (which can be automated) can handle that for flat files. You still need an appropriate target for it to backup to. A low power fileserver is a simple solution.

author: EEVblog
author_rank: Unknown
timestamp: May 26, 2013, 02:51:32 am
content: Quote from: Monkeh on May 26, 2013, 02:49:10 amWhy should I have to run my own box to do this when there are a zillion others already running 24/7?

author: Monkeh
author_rank: Super Contributor
timestamp: May 26, 2013, 02:53:56 am
content: Quote from: EEVblog on May 26, 2013, 02:51:32 amI don't know, because apparently none of them offer a reliable, secure service?I wouldn't trust siteautobackup.com as far as I could throw them, and seeing as they're 'cloud based' (ooh, shiny..) that's not very far.

author: EEVblog
author_rank: Unknown
timestamp: May 26, 2013, 03:13:25 am
content: Quote from: Monkeh on May 26, 2013, 02:53:56 amReally?I sense this is nothing but unsubstantiated fear mongering.Are you seriously suggesting that the Amazon S3 cloud that siteautobackup uses is less reliable than using a box running in my house?

author: EEVblog
author_rank: Unknown
timestamp: May 26, 2013, 03:16:55 am
content: Well, I just got the 504 error message again.This is now beyond ridiculous.I cannot believe that both capnel and WHM backups cause a dedicated server to lock up and refuse to respond every time it does a backup. If that was the case, millions of websites would be down.This same server used to work just fine using HostGators own WHM automated backup to a 2nd disk, and also using siteautobackup.comI suspect it's only since the server was switched over to 64bit and all the performance tweeks have been made that the problem has arisen. I could be wrong, but it most certainly used to work just fine with no reports of 504 gateway errors.

author: Monkeh
author_rank: Super Contributor
timestamp: May 26, 2013, 03:20:12 am
content: Quote from: EEVblog on May 26, 2013, 03:13:25 amI don't trust them from a security standpoint. Nor do I trust them to be doing their jobs right.If they provide unencrypted FTP access, they've failed completely already.

author: EEVblog
author_rank: Unknown
timestamp: May 26, 2013, 03:44:29 am
content: Quote from: Monkeh on May 26, 2013, 03:20:12 amThat of absolutely no help to me what so ever.QuoteThey don't, it has nothing to do with FTP.

author: Monkeh
author_rank: Super Contributor
timestamp: May 26, 2013, 03:46:42 am
content: Quote from: EEVblog on May 26, 2013, 03:44:29 amMy sincerest apologies for having an opinion.QuoteFair enough, that was me making 4am assumptions.Quote from: EEVblog on May 26, 2013, 03:16:55 amMillions of websites don't attempt to make a 7GB copy of themselves to another host every day. It's not a good way to go about things.

author: EEVblog
author_rank: Unknown
timestamp: May 26, 2013, 04:44:30 am
content: Quote from: Monkeh on May 26, 2013, 03:46:42 amSo what's your easy to use no fuss foolproof solution?

author: GeoffS
author_rank: Unknown
timestamp: May 26, 2013, 04:45:37 am
content: wilfred: I only run cPanel and not WHM.There's no facility for scheduling a backup in cPanel, Any backups you run from cPanel are created as gzipped tar files on the server. You can download these to your own server or cPanel can create them on another cPanel hosted system.Database backups are created as text files containing all the SQL statements to recreate the database from scratch, no facility for partial restore unless you know enough to edit the saved data.This is based on my own cPanel hosting on a shared server, might be different for dedicated servers

author: EEVblog
author_rank: Unknown
timestamp: May 26, 2013, 05:05:09 am
content: Quote from: wilfred on May 26, 2013, 04:27:12 amI'm using an external service siteautobackup.com that initiates initiates a scheduled backup and then download the file when it's finished.It then purges older backups and only keeps however many backup you tell it too in order to fit into the backup space you have paid for.I think that's a really neat solution, and I'd love to have something similar.It does the same thing for either Cpanel, WHM, or just the databases. Perhaps I can try just the databases? That would at least give me a back of them daily, and shouldn't take too much space or resources to do that?Then I could find another (incremental?) solution for the files?QuoteNo, cpanel is all or nothing.Quotecpanel is layer on top of WHM, which is a layer upon the basic linux install.According to the linux penguins it's evil to use either

author: EEVblog
author_rank: Unknown
timestamp: May 26, 2013, 05:10:14 am
content: Ok, another try with siteautobackup.comTrying a "partial" cpanel back right now. It is set to backup the databases and also all the home directory files. I can disable the files if needed, but thought I'd try it.Lets see if the server goes down again...

author: Monkeh
author_rank: Super Contributor
timestamp: May 26, 2013, 12:37:42 pm
content: Quote from: EEVblog on May 26, 2013, 04:44:30 amThere's no such thing, sorry.

author: xrunner
author_rank: Super Contributor
timestamp: May 27, 2013, 12:14:15 am
content: You're not gonna believe this, but it's my duty to report it. It lasted for at least 10 minutes.504 Gateway Time-outnginx

author: Unknown
author_rank: Unknown
timestamp: May 27, 2013, 01:32:45 am
content: Quote from: wilfred on May 26, 2013, 01:09:17 amThat's a fine way of doing non-intrusive backups. Another good way is to do the backup in a transaction (mysqdump --single-transaction), although this requires you to use storage engines with transaction support (eg. InnoDB). There will still be a performance impact, but at least the tables are not locked.

author: justanothercanuck
author_rank: Frequent Contributor
timestamp: May 27, 2013, 04:44:01 am
content: Does SMF store it's uploads in the database? 7GB seems like a lot, if it's just the text from our posts... FWIW, phpBB stores its uploads as files, making the database easier to backup.

author: EEVblog
author_rank: Unknown
timestamp: May 27, 2013, 07:52:42 am
content: Quote from: justanothercanuck on May 27, 2013, 04:44:01 amNo, its stores the uploads in a directory. IIRC the database is about 100MB or so last I looked.I've had limits set on the uploads for a long time, so thankfully it's only 7GB total for all the stuff, but it does grow daily.

author: justanothercanuck
author_rank: Frequent Contributor
timestamp: May 27, 2013, 08:02:03 am
content: Of course, thanks for the clarification.

author: metalphreak
author_rank: Frequent Contributor
timestamp: May 27, 2013, 02:09:10 pm
content: Does it put a lock on the files while it does the backup? How good is the connection from your server to the backup service? Just because you have a 100mbit port doesn't mean you have 100mbit bandwidth to every location. The backup could be taking forever at a slow transfer speed causing the 504's for a random length of time?

author: GeoffS
author_rank: Unknown
timestamp: May 31, 2013, 01:35:43 am
content: I just had another 504 occur. 11:30 local time.When I was again able to access the forum a couple of minutes later, no new posts were shown.I normally use the Show unread posts since last visit button but no posts were visible.

author: xrunner
author_rank: Super Contributor
timestamp: May 31, 2013, 01:38:20 am
content: I had the 504 error off and on for at least30 - 40 min. plus a couple of "SMF could not connect to the database ..." thrown into the mix.

author: EEVblog
author_rank: Unknown
timestamp: May 31, 2013, 02:02:29 am
content: Quote from: xrunner on May 31, 2013, 01:38:20 amYep, that was me, sorry.I have now narrowed it down.I tried to perform an optimisation of the database table, an option provided from within SMF, and got a 504 error straight away.But what cases the outage was that I when I tried a database backup (another option provided from within SMF) that immediately locked up the server with a 504 for at least 30 minutes. Something I have don many times in the past without issue.So it's not the backup the server files that is the problem, it's a matter of the database.gnif tells me that this is to be expected for a database of my size under cpanel/mysql, and the only solution is a MySQL slave on another server which is then backed up. Although I kinda find this hard to believe, as I've never had this issue before with either the former dedicated server box (32bit one, same host), cloud server, or shared server solutions. It seems to have only been since the move to the 64bit dedicated server that the issue has arisen. And the database hasn't gotten a huge amount larger since then, so that's what makes me think there could be something else at play here.But then again I know jack about this stuff, just going on my experience.

author: EEVblog
author_rank: Unknown
timestamp: May 31, 2013, 02:05:21 am
content: I'm about to tell siteautobackup.com to start a backup of the directory files only and none of the databases, I'm curious to see what happens...

author: EEVblog
author_rank: Unknown
timestamp: May 31, 2013, 02:13:55 am
content: Quote from: EEVblog on May 31, 2013, 02:05:21 amWell, the forum still works as fast as ever, so I guess automated file backup is not a problem, so far, backup not complete yet though. But at least it didn't lock it up.

author: GeoffS
author_rank: Unknown
timestamp: May 31, 2013, 02:27:21 am
content: From what you've reported, it sounds as if a database dump via (say) mysqldump is done with the tables locked. This would prevent any database access, read or write, until the backup process is finished. There are options that allow the dump program to operate reliably without exclusively locking the database.Is there any information available as to how siteautobackup handles the database for its backups?

author: EEVblog
author_rank: Unknown
timestamp: May 31, 2013, 02:30:34 am
content: Quote from: GeoffS on May 31, 2013, 02:27:21 amIf there is a such an option I'm keen to try it.QuoteNot that I'm aware of.The same thing seem to to happen in SMF or siteautobackup.

author: GeoffS
author_rank: Unknown
timestamp: May 31, 2013, 04:27:52 am
content: Another long 504 error session. Followed by a couple of SMF errors 'unable to connect to database'

author: EEVblog
author_rank: Unknown
timestamp: May 31, 2013, 04:37:14 am
content: Quote from: GeoffS on May 31, 2013, 04:27:52 amYep, sorry, likely me again.I did a databse backup from within cpanel and the site instantly went 504.Although I suspect the siteautobackup.com was still running as well.At least we know what causes it now!

author: edavid
author_rank: Super Contributor
timestamp: May 31, 2013, 05:19:19 am
content: Have you tried MySQLDumper?

author: ve7xen
author_rank: Super Contributor
timestamp: May 31, 2013, 06:36:08 am
content: Perhaps the table type has been changed in the move?I would suggest dispensing with mysqldump backups unless you have a replication slave.Shut down mysql to guarantee a consistent file structure, make an lvm snapshot (takes a few seconds at most), start mysql again and then duplicity/rdiff-backup or rsync your snapshot to a remote server. Duplicity is smart enough that you can use pretty much any file storage service, and both are incremental. A few seconds of downtime and a much faster (and less bandwidth intensive) backup process.You can use something like backupninja to help automate this (example), but it sounds like you're getting to the point where you need someone with actual systems experience to figure out the best way to handle all of this for your setup and implement it properly. It should be possible to do it with zero to minimal downtime and less bandwidth (and time) than your current solution.

author: GeoffS
author_rank: Unknown
timestamp: May 31, 2013, 07:57:23 am
content: Without knowing the current configuration and usage of the MySQL database(s), I think we're just guessing when it comes to suggesting better methods of backup.Standby sites for backup and failover are all well and good but someone has to pay for the necessary resources.

author: EEVblog
author_rank: Unknown
timestamp: May 31, 2013, 10:06:41 am
content: Quote from: wilfred on May 31, 2013, 07:42:19 amThe world won't end if it takes me a day or two to fix it. I'd lose incoming email in that time, but no biggie.QuoteThe current dedicated server is pretty overkill. CPU load for examples borders on trivial.gnif has figured out it's the persistent connections in the fourm software in combination with the myqsl version and cpanel backup etc. He's currently trying to patch it.So this is hopefully close to being fixed.

author: arekm
author_rank: Unknown
timestamp: May 31, 2013, 11:15:00 am
content: Quote from: EEVblog on May 31, 2013, 02:02:29 amYou don't need a separate server. You can run two mysql instances on single machine, one as master, one as slave and then do backup from slave. Unfortunately no such feature in cpanel as far I know. Tiny amount of scripting (well, script for starting new instance actually) would be required.By default mysqldump locks tables for write operations, so it can get consistent snapshot of database. There is option to turn that off but that also means loosing consistency.Also which engine is used for tables - innodb or myisam? Innodb is superior with things like row locking (instead of table lock) etc. Innodb won't help with backup issue but for the other things (like just running forum) it's much better than myisam.

author: gnif
author_rank: Unknown
timestamp: May 31, 2013, 12:26:41 pm
content: Sorry for not getting back in here to let people know what is going on... been busy trying to track this down.Running a MySQL slave on the same host is a bad idea, granted it works, but think about it... the server is experiencing high load issues during the backup from the master... if you throw a slave on there and perform a backup from the slave, firstly you will be wasting server resources as MySQL will be trying to keep everything in RAM for both servers. Secondly this is a hack to try to work around a problem that is not fully understood, the correct thing to do here is more pervasive testing and diagnostics, as Dave has said countless times, why has this occurred since the move from the old server, what has changed?It was discovered today how to reliably trigger this error which exposed the persistent connection issue.MySQL persistent connections were part of the problem, but not the solution, there were threads getting into deadlocks leaving tables locked when the MySQL dump process ran... this then caused PHP processes to get stuck waiting for queries to execute bogging the server down to an unusable point.This was caused by the deprecated PHP MySQL extension as it does not handle persistent connections with later MySQL versions (This is clearly documented on php.net). There were two options here, 1) move to MySQLi, which there was not a suitable mod for SMF do do this, or 2) drop using persistent connections (would we prefer not to), so I wrote a MySQLi driver for SMF and is working perfectly.Now I could try to dump the data again, this time doing so ran perfect, the CPU load stayed down at 0.8 and I did not see a million PHP processes spawning, nor deadlocked tables. So that issue is fixed... but a new issue came up.The server started to run out of RAM until it eventually did and started swapping to disk. Further investigation showed that one of the SMF updates done at some point had converted some of the tables to InnoDB and MySQL was trying to cache a lot more data then we were expecting it to. Calculations showed that MySQL would try to consume a total of 3.8GB of RAM with the settings it had been configured with, so I went to work on tuning this and dropping its total RAM usage.This involved firstly dropping the amount of ram each thread could have for temp tables, joins, sorts, etc. after which I logged slow queries that were not indexed properly or were poorly written. After fixing the indexes on about 8 different tables, the memory requirements for the queries went down somewhat and performance has improved, but this only brought us down to 3.1GB usage, only leaving 900MB for other things, which is not enough.So we are finally at the crux of the issue... we need some more RAM, 4GB additional should do it, but the more the better as we can cache more in RAM for better performance of the site and forums.In short, the problem was a whole heap of things compounded together. People need to understand that there are a million ways to configure MySQL for performance/caching etc... your workload dictates how it needs to be tuned and as such there is no single simple solution to an issue like this.I do know there is more queries being executed by SMF that are complete and utter trash, such as INDEXOF queries looking for PHP serialized data in text fields , this can NOT be indexed and forces a full table scan every time the query is executed. This is a good example of programmer meets database and thinks, it works good enough on my PC/Server so it must be good, but until it gets scaled to large environments they have no idea how poor the idea is.So for those developers, if you need to search on data in a serialized text field, DON'T SERIALIZE IT, don't be lazy, add the extra columns and store the data properly, and add an index to the column. Serialized data is only useful if you don't need to search on it, if you cant get away from that fact due to design constraints perhaps it is time to consider a different design using a NO-SQL engine such as MongoDB.

author: gnif
author_rank: Unknown
timestamp: May 31, 2013, 12:37:29 pm
content: Oh, also I had made the incorrect assumption that Dave's site was on a shared environment which a huge amount of hardware behind it, which is usually the case in large shared environments, this is why I was convinced that it was due to the site being moved to a less capable server in terms of I/O throughput.Also for some reason I thought that this server was on a RAID10 array, which caused some concern about the throughput of the array being way too slow for a RAID10 array. Dave thankfully corrected me here, its on a RAID1 array (clone).

author: GeoffS
author_rank: Unknown
timestamp: May 31, 2013, 12:45:25 pm
content: gnif,Well done for tracking this down.

author: gnif
author_rank: Unknown
timestamp: May 31, 2013, 12:50:10 pm
content: Quote from: GeoffS on May 31, 2013, 12:45:25 pmThanks, now just to track down that bug on your avatar! I hate that thing!

author: SeanB
author_rank: Super Contributor
timestamp: May 31, 2013, 03:23:55 pm
content: Login as admin, look up user and replace the file..........Just saying, no need for the fly swatter, though it does drive the mozzies here mad as they try to mate with it, and blood is so hard to get off a LCD.

author: GeoffS
author_rank: Unknown
timestamp: June 03, 2013, 12:42:22 am
content: FYI Another 50410:40 local time.

author: xrunner
author_rank: Super Contributor
timestamp: June 03, 2013, 12:44:59 am
content: I was getting a blank screen too - no text, no graphics. Just a plain white screen. Don't think I've ever seen that trying to access any forum before.

author: EEVblog
author_rank: Unknown
timestamp: June 03, 2013, 12:56:19 am
content: I just got a site down report about an hour ago, 10am sydney time. No idea why.Huge thanks to gnif for tracking it down.An extra 4GB of RAM will cost me an extra $100/month, yes, a month, forever. Such is how Hostgator charge.No, I will not consider changing hosts again, I do not want the grief, nor to rock the boat, this server has been very good.gnif, will disabling persistent connections fix the memory problem?I know you don't like it, but the server has worked perfectly before without it.

author: gnif
author_rank: Unknown
timestamp: June 03, 2013, 03:54:42 am
content: Quote from: wilfred on June 03, 2013, 02:16:26 amNot at all related to the issue any more.. persistent connections do not use more RAM, if anything they use less.MySQL as any good database will, tries to cache as much into RAM as it can to increase query performance, it has nothing to do with how many users are connected or how many queries it is running.In general day to day use on the forum, only a subset of tables/records are being hit, and they are slowly getting cached on the fly as people view posts/threads. Inevitability this will fill up the RAM on the server simply because there is not enough ram.When the backup process runs, MySQL takes advantage of the fact that the dump is reading every single record, so it caches it up to MySQL's configured limit into RAM to avoid having to hit the hard disks as often. This cache is primarily the indexes.Reducing the cache able size is not the answer, if there is not enough cache room then MySQL performance suffers greatly, which will also cause backed up PHP processes waiting on slow queries, causing the 504 error again because the PHP back-end stops responding.The modifications to SMF are future compatible, Dave can seamlessly upgrade without problems. The new code is a separate file/class that is just configured in the settings to be used instead of the default 'mysql' driver. There is already support for multiple drivers (pysql, sqlite, etc), I just added a new one (mysqli).Dave's database backup is NOT 7 GB, that is the database + website files... the database is only 1.1GB compressed when dumped out.In an attempt to keep the server operational I have detuned MySQL to only cache up to 50% of the system RAM as opposed to the 113% it was previously configured to take. The outage today was caused by MySQL running out of RAM room to cache records and queries running very slow, as described above.The only solution here is more RAM, or delete 1/2 the database so MySQL does not try to cache so much.And for those that are saying that InnoDB is better then MyISAM, you clearly have no experience in large enterprise environments, MyISAM has its place still and can often be MUCH faster then InnoDB, it just depends on the workload and if you need row-level locking for what you are doing. InnoDB also requires more RAM, as is the original cause of this issue.In this case the better engine for the posts table is InnoDB as row-level locking is better then table level locking, and this is why it has been converted by later versions of SMF, but many of the tables are still MyISAM simply because it is faster for what they are being used for.An example of a different workload that InnoDB sucks at:SELECT COUNT(*) FROM my_table_with_100000_rows;This can take up to a minute to run, or longer depending on the column count, where MyISAM returns this in 0.001 seconds as it keeps a cache of the row count.

author: gnif
author_rank: Unknown
timestamp: June 03, 2013, 06:14:43 am
content: Quote from: wilfred on June 03, 2013, 05:46:09 amMySQL was never able to cache it all, that is the problem, and why it was performing poorly. If it was able to cache properly the hit ratio would be 100%. With the usage reduced to 50% of system ram (which BTW is still too high when you take into account the other processes on the server) the hit ratio is a dismal 20%.Uncompressed the dump is smaller then the actual database size because it does not contain the indexes... these are re-built when the import occurs. The database is 3.1G uncompressed in binary form including indexes. The indexes are stored on the disk in compressed form to save disk space and improve read speed, when they are read into ram they consume even more space.Quote from: wilfred on June 03, 2013, 05:46:09 amThe database is seeing 190 to 250 queries per second on average, and I have seen this peak out at around 1,200 queries per second on days that Dave releases new videos, or days that the site has been published on HaD. Each user that is new to the site is more likely to visit pages that have not been hit in a while, which pull data from the databases that is not cached. If the database did not cache as much as it does the site would have ground to a halt during these peak periods.For more information on how MySQL caches InnoDB data please seehere.Quote from: wilfred on June 03, 2013, 05:46:09 amDon't worry, I did not take it that way. It is not my driver that handles the clean-up, it is the newer PHP MySQLi extension. My 'driver' is really just a database abstraction layer between SWF and whatever database you want to use. If you dig through the changelogs for MySQLi you will see they fixed persistent connections by calling mysql_change_user after each session which is a MySQL API that resets the connection state, including locks, transactions, etc. It introduces a little bit of overhead but not as much as reconnecting each time.The only cache that is connected to a session is that of a transaction, table sorts or joins... and once mysql_change_user is called when the script shuts down, all of this is released. Also it is configured to only allow 3MB per connection of cache before it makes a temp file on the disk.Also, another point to note, table joins, and and ORDER BY result sets require comparably large amounts of ram when compared to simple SELECT query. Each connection is allowed to use up to tmp_table size of RAM for these queries. When the tmp_table size is exceeded it starts to write and sort the results on the disk, which needless to say is very slow. Unfortunately SWF has a heap of these bad queries and as such needs a large tmp_table buffer to keep the site humming along. At the moment it is very low, which is causing MySQL to swap to disk way too often (41% of queries requiring a temp table are being written to disk at present).

author: gnif
author_rank: Unknown
timestamp: June 03, 2013, 06:31:17 am
content: Here are the RAW stats from the database in its current configuration:Data in MyISAM tables: 474M (Tables: 411)(This is DATA only, not indexes)Data in InnoDB tables: 1G (Tables: 381)(This is DATA only, not indexes)Total buffers: 1.5G global + 3.9M per thread (150 max threads)Maximum possible memory usage: 2.1G (54% of installed RAM)Key buffer size / total MyISAM indexes: 512.0M/203.5M(These are not InnoDB buffers)Key buffer hit rate: 99.9% (314M cached / 282K reads)(These are not InnoDB buffers)Query cache efficiency: 83.2% (30M cached / 36M selects)Query cache prunes per day: 349756Sorts requiring temporary tables: 43% (43K temp sorts / 100K sorts)Joins performed without indexes: 49662(Unavoidable without rewriting chunks of SMF)Temporary tables created on disk: 41% (50K on disk / 116K total)(This is one reason we need more RAM)Thread cache hit rate: 99% (1K created / 1M connections)Table cache hit rate: 99% (8K open / 8K opened)Table locks acquired immediately: 99% (13M immediate / 13M locks)InnoDB data size / buffer pool: 1.7G/1.0G(This is not great, but not the end of the world)

author: justanothercanuck
author_rank: Frequent Contributor
timestamp: June 03, 2013, 12:00:47 pm
content: Back in one of the other threads, Dave said the forum was only using half the server's physical memory... can't you just increase tmp_table_size?

author: gnif
author_rank: Unknown
timestamp: June 03, 2013, 12:06:32 pm
content: Quote from: justanothercanuck on June 03, 2013, 12:00:47 pmAnd the database is using the other 1/2 already, how can I increase it when there is no more physical RAM for it to use?

author: justanothercanuck
author_rank: Frequent Contributor
timestamp: June 03, 2013, 12:15:29 pm
content: Sorry, I don't mean to sound like a knob, it's just that I've run and used other high-use forums that work with less hardware and not have these problems.

author: xrunner
author_rank: Super Contributor
timestamp: June 03, 2013, 12:27:33 pm
content: Quote from: justanothercanuck on June 03, 2013, 12:15:29 pmYou know that's what I've been thinking all along this thread. I go to some high-use forums, one of which I know for a fact uses the same shared server that my personal forum is on (which has few users), and I've never seen a 504 gateway time out on that forum, nor has anyone done the slightest bit of "tuning" of anything.I do know that you should not do backups from within the Admin panel even though it has that function. Here's what SMF itself says about it -QuoteBut an even easier way is to just go to Backups in Cpanel and use that. Just use "Download a MySQL Database Backup. It's automatically compressed too. I've never seen it disrupt the forum. I've restored it from that backup and it worked perfectly.

author: gnif
author_rank: Unknown
timestamp: June 03, 2013, 12:43:43 pm
content: Quote from: justanothercanuck on June 03, 2013, 12:15:29 pm1) That screenshot is pre-migration, which is before SMF converted to InnoDB tables, and running default cPanel MySQL config which performs extremely minimal database caching.2) The server is not just running a high use forum, we also have wordpress on there.3) Every tuning trick in the book has been attempted, believe me, I have spent more hours then I care to count trying to find a resolution to this before falling back to recommending that hardware be upgraded.4) You may be running high use forums, but it does not mean that the server workload is the same... do you run SMF? what version of PHP? what version of MySQL? are you running memcached? are you running APC? how much ram does your server have? Are you on a RAID10 array or using SSD?, or a VPS that has a SAS array as it's backend? I could go on... there are many many variables here that can contribute to this problem.Here it is in short...SMF upgrade caused table conversion to InnoDBServer migration moved from PHP 5.2 to PHP 5.4Server migration moved from MySQL 5.0 to MySQL 5.5Old server was prone to SMF errors at random, but less often then new server (aka: pre-existing problem).Hours of investigation, testing, profiling and evaluation proved that the deprecated MySQL extension + MySQL 5.5 was causing deadlocks which was fixedFurther hours of investigation, testing, profiling and evaluation proved that the server is simply out of RAM when it comes to caching the larger record set required for tables using the InnoDB engine.Database was de-tuned to be suboptimal to prevent outages caused by excessive RAM usage, this included reducing temp table sizes, innodb buffer pool sizes and ram used by each thread. This introduces a NEW problem, where MySQL is not returning to PHP as fast as it should on certain queries (forum searches for example).Lets say a user loads the forum and performs said search... this query loads down MySQL as it is intensive and takes say an extra second or two.We are seeing an average of 200 queries per second.... that means that for a full second, 200 said queries are either slower, or blocked by locked tables.That means that every other visitor to the site gets served slower... which means that each PHP process serving an individual user is running for longer, holding onto more RAM.More visitors hit the server while it is still trying to play catchup with all those other HTTP requests... they also ask for database info... they also get held up...The result is, the small problem of some slow queries causes the entire server to snowball into oblivion.QuoteThat is because we are running a reverse proxy that has time-outs on how long to wait for the back end to respond, if that time out is hit, you get a 504. Without the reverse proxy you get users sitting waiting for the page for as long as they want to wait for it to load, which not only makes them think the entire server is broken, but increases the load on the server enormously each time they hit 'refresh' trying to make it work. The difference here is you are getting a real error, not just a blank white screen while you pray that the site is going to load.QuotecPanel uses mysqldump and gzip to dump it's database out... we are not using SMF to backup the database. Using any PHP based database dump utility is a stupid idea, as most of servers have PHP configured with max_execution_time of 60 seconds to prevent bad scripts from running too long and pulling the server offline.You said this is on a shared server.... so you have no idea how much RAM the machine has, nor what kind of storage it is using, it could be a RAID10 SSD array for all you know. It may even be connecting to a dedicated database server that does nothing but handle the database.

author: gnif
author_rank: Unknown
timestamp: June 03, 2013, 12:55:54 pm
content: And one more thing that changed between servers, 32->64bit... the system across the board will consume more RAM as it requires 64bits for memory addressing instead of 32bits, this is a commonly known fact (see here for an example). This is why low memory VPS servers are recommended to run 32bit operating systems to conserve memory for applications.

author: EEVblog
author_rank: Unknown
timestamp: June 03, 2013, 12:56:16 pm
content: Quote from: justanothercanuck on June 03, 2013, 12:00:47 pmHere is the latest one on the new 64bit server. It uses less memory than before.I'm not sure why the backup using the RAM can't just swap to disk for a while and the forum runs slower for a little while during the backup process?In the screen shot below the server seems to be using 11% swap anyway?

author: xrunner
author_rank: Super Contributor
timestamp: June 03, 2013, 12:56:35 pm
content: Quote from: gnif on June 03, 2013, 12:43:43 pmWell gnif, if it's a stupid idea then the entire support staff of the very software you are using for this forum (SMF) are utter idiots, because that's the advice they give to people.

author: gnif
author_rank: Unknown
timestamp: June 03, 2013, 01:08:19 pm
content: Quote from: EEVblog on June 03, 2013, 12:56:16 pmThe dump process is not caching or using the ram... MySQL is. I don't know how else to say this. The forums do run slower when it swaps out to disk... but, it runs enormously slower. 200 queries per second start getting backlogged, it does not take long for this to pull the server down.As for those readings... they are not taking into account what is cached, a more accurate readout would be from the free utility on the command line:Code:root@eev [~]# free -mtotal used free shared buffers cachedMem: 3959 3779 180 0 180 2005-/+ buffers/cache: 1593 2366Swap: 4095 470 3625Your true memory usage according to the system is (100 / 3959) * 3779 = 95.5%. Since MySQL flags its cache as 'SWAPPABLE' it is included in the buffers/cache value which cPanel is not counting.Swap usage occurs when the kernel swaps something out when there was not enough ram at some point and an application has not requested access to it again. As soon as it is wanted again, if there is enough free RAM it will be read back into RAM out of swap. If there is not enough RAM to read it back in, the oldest object in RAM will be kicked into swap to make room.Quote from: xrunner on June 03, 2013, 12:56:35 pmThey give that advice because the general population are not server administrators and do not know how to perform the backup, or do not have the server access required to perform a mysqldump, I am in no way calling them idiots.Have you not noticed that a cPanel backup is a process that runs in the background, and it emails you when it is done. This is because if the dump takes too long, or you close your browser, or the script times out, or a proxy between you and the server crashes/closes the link, etc... you will never get your backup.While what they do works, and I have had to use it on occasion with other PHP based dumps, it is not recommended on large datasets.

author: justanothercanuck
author_rank: Frequent Contributor
timestamp: June 03, 2013, 01:15:39 pm
content: Quote from: gnif on June 03, 2013, 12:43:43 pmMy point exactly.The forum I was referring to was at one point, one of the largest linux distro forums. We were averaging one new user every 5 minutes, thousands of concurrent users, all running on a single server with no errors. In fact, most of their problems started when they tried going from a single server - to two web servers, two db servers, and two load balancers. Now they average less than 500 concurrent users, and get blank pages and strange errors at all times of the day. I won't even get into the hassles and apathy I received when I attempted to upgrade them from phpBB v2 to v3, which to this date, they still haven't started to attempt (which was why I quit).I'm not saying you're not trying, and I wouldn't even start to suggest otherwise, I'm just saying something definitely isn't right if we're bouncing off limits all the time.

author: gnif
author_rank: Unknown
timestamp: June 03, 2013, 01:17:33 pm
content: Quote from: justanothercanuck on June 03, 2013, 01:15:39 pmI will say again, what did they move from? How did they tackle issues with the new server? did they just throw hardware at it or did they profile and diagnose the problem?Edit: Also, here are the stats on another server I manage that runs a phpBB forum:Threads: 634,533, Posts: 11,528,628, Members: 129,996, Active Members: 10,457This server runs 24x7 including backups using MySQL dump, flawlessly. To my client it appears to be a single cPanel server, but in fact it is two separate servers, the cPanel server and a Debian database server with 8GB of RAM configured to keep as much in cache as possible. The site performs better then the previous configuration they had which consisted of 4 back end servers with a reverse proxy balancing requests between them, simply because the previous admin had no idea how to profile the application and locate the bottleneck, nor how to tune a MySQL server.

author: xrunner
author_rank: Super Contributor
timestamp: June 03, 2013, 01:23:16 pm
content: Quote from: gnif on June 03, 2013, 01:08:19 pmYou said it was a "stupid idea". Idiots give out stupid ideas, not helpful knowledgeable support people.QuoteI've never gotten an email when mine was done. As far as whether it's a background process or not I couldn't care less. It works and has for years.QuoteI can't confirm this as I've never not gotten my backuo.QuoteMy forum has ~195k posts. This one has ~235k posts - essentially about the same in terms of database backup size. My database backup takes only a minute or so and has never failed and I've never not had it complete and it's on a shared server.Many, many people use SMF forums, and some for large busy forums, and they don't need all the tuning and knowledge you have to make them work well and get reliable backups. I've really never read of an SMF forum owner on the support forums that had to muck around with all the little things you seem to be having to twiddle. Not that it's never happened I've just not read of it. If the level of twiddling you guys are having to do was needed all the time then hardly anyone would ever have a successful busy forum, and I know for a fact that they do without all the twiddling.I'm only relating the experience I've had with SMF forums for all the years I've been using and installing them and reading on the SMF support forums, I'm only trying to help.Have you registered at the SMF support forum? If so why don't you ask for help from them - they're the experts after all.

author: arekm
author_rank: Unknown
timestamp: June 03, 2013, 01:27:08 pm
content: Quote from: gnif on June 03, 2013, 12:43:43 pmFor search purposes using sphinx (http://sphinxsearch.com/) is great. Here searching ~50GB (dump compressed with xz is ~16GB) mysql database takes ~5s.

author: madires
author_rank: Super Contributor
timestamp: June 03, 2013, 01:34:49 pm
content: Quote from: EEVblog on June 03, 2013, 12:56:16 pmJust 4GB RAM? All the web sessions with PHP require a lot of RAM already. IMHO a two server setup would be better, i.e. one server running the webserver and the forum and a second one for the database. I don't know the size of the database but I guess 8 or 16GB RAM just for the database should improve things considerably. And the database is increasing with every post. If you go for such a solution ask your hoster to add an additional NIC to both servers and connect them back-to-back (crossover) just for the database traffic.

author: gnif
author_rank: Unknown
timestamp: June 03, 2013, 01:37:16 pm
content: Quote from: xrunner on June 03, 2013, 01:23:16 pmPoint taken, will be a little more careful with my wording in future.Quote from: xrunner on June 03, 2013, 01:23:16 pmIt is optional, you can enable it when you choose to perform the backup. It also depends on if cPanel has been configured to properly relay to itself, although I have never seen this broken by default. My point about the backgrounding, it means that if for whatever reason you loose your connection, there will still be a backup file waiting there for you to download when you come back, you wont have to trigger another backup.Quote from: xrunner on June 03, 2013, 01:23:16 pmAgain, read what you are saying, it is on a 'SHARED SERVER', theses servers usually have 10x the budget thrown at them... they often have dedicated database servers. The ROI on theses servers warrant that kind of investment in hardware. Anyone running a shared server with less then a RAID10 array and 8GB of ram is kidding themselves. Find out what the hardware is in your shared server before you compare it to this one.Quote from: xrunner on June 03, 2013, 01:23:16 pmI understand you are trying to help, and I appreciate it, but what you do not get is the problem has been located, the cause has been located, and there is a solution to the issue. And for the record, the default MySQL tuning on a cPanel server sucks ass, all configuration settings are left default and is the reason why most cPanel servers do not deal well with large sites until the admin either adds RAM and tunes it, OR starts using a dedicated database server.Edit: You also state that you 'install SMF forums', but have you ever been the server admin that monitors the machine and throws more hardware into the mix when you notice that there is a performance drop across the board? On this server, my hands are tied as Dave has a limited budget for the server, I am already offering him my enterprise level support for free to help him keep this site up and running. This is not my first time with this kind of an environment, nor will it be my last... it is however the first time the community has been so involved in what is going on at a technical level.Quote from: arekm on June 03, 2013, 01:27:08 pmThe issue of the search was an example, it is one of the many queries that are logged as being slow, the other one is this horrible join of 4 tables which includes a string search on a php serialized array for encoded data. And unless SMF has built in support for sphinx, there is no point bothering as the upgrade path for Dave needs to be kept standard (ie, no modifying files that will get overwritten during an upgrade).Quote from: madires on June 03, 2013, 01:34:49 pmMadires, I would LOVE to set this up, but the issue is the cost.

author: justanothercanuck
author_rank: Frequent Contributor
timestamp: June 03, 2013, 01:50:14 pm
content: Quote from: gnif on June 03, 2013, 01:17:33 pmThe original server (2003-2004?) was a dual processor P4 2.4ghz, with 4gb of memory. Their current web servers (2007-2008) are 4x 2.4GHz Opteron 2216's, with 8gb of memory. Their db servers are 8x 2.3GHz Opteron 2376's with 16gb of memory. No idea what their load balancers use. Their database is 24gb in size, with compression disabled.So essentially, they did just throw more hardware at it, but they didn't have issues until the original server had a hardware failure and they replaced it with the hodgepodge they have today. They run an optimized phpBB v2, or so they claim, but v3 has most of their changes and would run better if they ever choose to upgrade.

author: EEVblog
author_rank: Unknown
timestamp: June 03, 2013, 01:52:43 pm
content: Quote from: justanothercanuck on June 03, 2013, 01:15:39 pmgnif will no doubt correct me if I'm wrong, but as I understand it, there is nothing wrong with the forum itself, it is lighting quick. It's the backup process that is causing the problem. At the moment I have disabled automated backups, and have no way to generate a backup without bring the server down as we've been seeing.gnif, is the database slave on another server still a workable solution?I'm surprised there is not some "slow" way to backup the database. e.g. suck the database contents slowly over a day or something. After all, there are already 200 database connections per second every second?Yes, I know it wouldn't be an instant snapshot, but with currently 600+ new posts a day, any server crash will ultimately mean loss of at least some recent data anyway.I should just pony up for the extra ram, but I just hate the way Hostgater gouge for this

author: madires
author_rank: Super Contributor
timestamp: June 03, 2013, 02:04:44 pm
content: Quote from: gnif on June 03, 2013, 01:37:16 pmUnmanaged server in a German data center for €69/month:- Xeon® E3-1245 Quad-Core- 16 GB DDR3 RAM ECC- 2 x 3 TB SATA 6 Gb/s HDD 7200 rpm (Software-RAID 1) Enterprise-Class- 1GigE NIC and 100FE uplink port- 10TB traffic inclusive- 100GB backup space- reset service, rescue system and some basic stuff moreManaged server (same hoster as above) for €99/month:- Core™ i7-2600 Quad-Core- 16 GB DDR3 RAM- 2 x 3 TB SATA 6 Gb/s HDD 7200 rpm (Software-RAID 1)- 1GigE NIC and 100FE uplink port- 10TB traffic inclusive- updates, security package, daily full backupAnd you can buy hardware and service options for some bucks more. They also do custom stuff.

author: xrunner
author_rank: Super Contributor
timestamp: June 03, 2013, 02:04:50 pm
content: Quote from: gnif on June 03, 2013, 01:37:16 pmSo? I'll just do another backup - it doesn't take that much time.QuoteOK.QuoteSo - what are we all talking about then?QuoteI've never had a problem with it, but if you say so then it must be so.QuoteNo, I don't need to know all that, that's what the hosting company is paid to take care of.I don't know how to build an oscilloscope either, but I can damn sure tell if it's working right or not. It's up to the oscilloscope maker to manage the design of it for the end user just as it is for the hosting co. to manage the design of the server setup for the end users.QuoteI want this place to be a success too.I commend you on offering your support, but it appears you guys have reached a crossroads. Logically you either have to find the budget for the services you are attempting to offer (on the server), or, reduce the level of services you are offering (on the srerver).Is there a third option gnif?

author: madires
author_rank: Super Contributor
timestamp: June 03, 2013, 02:12:53 pm
content: Quote from: EEVblog on June 03, 2013, 01:52:43 pmmysql can be set up in master/slave or cluster mode. If you setup a slave on another (low-end) server all data will be synchronized as it is added/changed/deleted on the master. That would be a time distributed backup.

author: gnif
author_rank: Unknown
timestamp: June 03, 2013, 02:42:19 pm
content: Quote from: EEVblog on June 03, 2013, 01:52:43 pmYes, this is pretty much correct, the configuration of MySQL and PHP is perfect for the forums, that is why they are so fast.Quote from: EEVblog on June 03, 2013, 01:52:43 pmUnfortunately no, I suggested this before I discovered what the actual problem was, even if you move the backup process to another server, this site will still crash every X days as MySQL gradually fills its buffers with data from the disk.Quote from: EEVblog on June 03, 2013, 01:52:43 pmIt is not about the speed of the backup, to take a dump of your database you need to take one that is consistent. Lets say for example we have two tables.Users and Posts.You start the backup, which commences dumping out users... lets pretend this table is huge and it takes a few minutes. While that backup is occurring, a new user has registered and has posted a few things. Because your backup of the users table started before this user registered, it will not contain the new user registration. Then your backup starts to dump the posts table, which contains the new posts created by that user.Now, if you had to restore this backup, you would have what is called 'orphaned records', because the parent that they reference, (the user) is missing. This can cause all sorts of weird behaviour in the application using the database, ranging from missing text on the page, to a complete crash of the forum when someone tries to view the records, it just depends on how well/poorly the application handles the error.To get a consistent dump you need to lock all the tables so that any writes to them gets blocked until you have finished your backup. If you made your backup slow, it would mean that nobody could post new data to the forum for as long as the backup takes.Now, let me try to explain what MySQL is doing and where this RAM is getting consumed.RAM 101:Memory is extremely fast compared with all other forms of storage, but it is also extremely limited. Computers use the RAM for temporary storage, but when a computer runs out of ram, in the old days you just got an 'Out of memory error' and your program would crash. So swap space was invented.Swap 101: When your RAM fills up and an application wants more... the kernel of your operating system looks for a chunk of memory that has not been accessed in a long time and kicks it over to the hard disk. It then gives that 'freed' space to the application that wants it.Linux + Swap: Every time a file is read from the hard disk, Linux keeps a copy of it in RAM if there is room for it, this is why memory usage under Linux can read high even if you are not doing anything with your computer. It does this because ram is hundreds of times faster then your hard disk. The next time an application on your PC requests access to the same file, Linux will just grab it from RAM, it wont go to the hard disk again. If this is a file cached by the kernel from an operation such as a file read, it will NOT swap it to disk, it will just discard it.Linux memory model: When an application asks for some RAM, it can specify to the kernel if it is allowed to swap it to disk or not, and it can specify if it is a buffer or not. If the memory is flagged as non-swappable and there is not enough free physical ram, the application behaves like there is no swap file and gets an 'Out of memory error'.MySQL: The MySQL developers also know that RAM is faster then the hard disk, and is perfect for performing search and sort operations from. If they wanted to they could make MySQL read it's entire database into RAM at start-up, which would mean that MySQL would take quite some time to start, and would hold lots of crap in memory that was very rarely accessed, if at all. So MySQL works like the kernel, as data is read from the disk to hand it to a client, be it a website or a program that dumps the database for backup, it keeps a copy of the data in RAM. The more unique data that you read from the database, the more RAM it will consume.MySQL does not know how much RAM it has to use, it is configured in my.cnf through various settings. If improperly configured it will attempt to use more RAM then is physically available at which point the kernel jumps in and starts kicking things into the swap file. This is where it gets bad... as now you might as well have NO fast ram and be running entirely off your hard disk.I hope this clears it up a bit for you Dave.Quote from: xrunner on June 03, 2013, 02:04:50 pmFor you yes, but on larger systems, or slower systems, it might take an hour. The point is that it is not a good general recommendation to use a PHP based MySQL dump utility.Quote from: xrunner on June 03, 2013, 02:04:50 pmIf you never manage the server you wont know how it is configured, a user on a shared server never gets exposed to this level of the configuration. Just Google 'cPanel MySQL tuning' for an idea of how many people have to fix this up. This is no fault of cPanel btw, there is no way for them to know what kind of server cPanel is going to be installed on, nor what kind of work you are going to have it do, you might not even be using the database.Quote from: xrunner on June 03, 2013, 02:04:50 pmExactly, and you are talking to one of those guys you pay to take care of those things.Quote from: xrunner on June 03, 2013, 02:04:50 pmThere is no third option unfortunately, we are at the point where Dave has to make a decision.

author: xrunner
author_rank: Super Contributor
timestamp: June 03, 2013, 02:51:18 pm
content: *** This post is for a reference data point only ***I just backed up the database on my forum which has 195,000 posts. It took only 68 sec. It resulted in a 58 Mb compressed file.Your forum has 235,000 posts (I realize the size of the posts may be bigger or smaller than the ones on my forum). It should not take much longer to back it up though. I realize the servers are different and the setups are different - but a realistic time period to back up your database would be under two minutesgiven the needed resources*** This post is for a reference data point only ***Quote from: gnif on June 03, 2013, 02:42:19 pmWell, we agree on that.

author: gnif
author_rank: Unknown
timestamp: June 03, 2013, 03:03:13 pm
content: Quote from: xrunner on June 03, 2013, 02:51:18 pmI have timed the database backup on Dave's server when it has the RAM available to do it successfully, and it takes a total of 2 minutes. The machine is plenty fast enough, it just needs RAM so it can do this while the site is in use.

author: Unknown
author_rank: Unknown
timestamp: June 03, 2013, 03:13:41 pm
content: How about just shutting httpd down for a few minutes at the time with the least traffic? I realize that there are no well defined peak times as on a national site, but some times have got to be busier than others. A few minutes of downtime sucks, but it would be better than no regular backup, and I imagine 0.2% downtime isn't worth $100/mo to Dave. You could show a well defined error page, so people don't complain to Dave, and the backup will probably finish quicker if it doesn't have to handle all the other traffic.

author: gnif
author_rank: Unknown
timestamp: June 03, 2013, 03:16:41 pm
content: Quote from: alm on June 03, 2013, 03:13:41 pmQuote from: gnif

author: xrunner
author_rank: Super Contributor
timestamp: June 03, 2013, 03:19:45 pm
content: Quote from: gnif on June 03, 2013, 03:03:13 pmThat sounds reasonable alright.Well I'm sorry that the extra RAM/Month is so expensive, but painful as it is, you may have to find a more reasonable host in terms of price and move to it. If that's not an option, then as we agreed, logically something else has to change.

author: Unknown
author_rank: Unknown
timestamp: June 03, 2013, 04:02:43 pm
content: Quote from: gnifThen memory is being oversubscribed: MySQL is allowed to use more memory than available. What's the point of having larger buffers that you can't use without swapping to disk anyhow? If it doesn't hit swap until X days, wouldn't lowering the limits be without performance impact for the same X days? MySQL taking care of the low memory condition by purging old stuff from its buffers will be more graceful than the kernel doing the same.I admit it's been a few years since I managed large(ish) MySQL databases (this doesn't really qualify as large), but I remember MySQL having fairly straightforward memory settings, so for example lowering the innodb_buffer_pool would reduce the memory usage by a similar amount. The main variable is the number of per-connection buffers, which varies with the number of connections.I agree that 4GB is quite anemic for a database + web server, but if it performs fine for a few days, than tuning it so it always performs fine should be possible.

author: gnif
author_rank: Unknown
timestamp: June 03, 2013, 04:06:32 pm
content: Quote from: alm on June 03, 2013, 04:02:43 pmAgreed, and this is why I have pulled it's memory usage right back and this forum still runs quite well. The issue is that it is sharing ram with PHP, eventually someone will upload a large image to resample, PHP will consume RAM that would normally be available for MySQL and even with it's reduced memory footprint it will still swap to disk and crap out. The issue is that PHP is sharing the ram with MySQL. MySQL expects a fixed amount where PHP can consume pretty much as much as it wants at any time.

author: Unknown
author_rank: Unknown
timestamp: June 03, 2013, 05:04:56 pm
content: Yeah, I was considering that issue. Spikes in memory usage are annoying. I don't think Linux has very good partitioning features, apart from heavy handed solutions like kvm.Limiting memory with mod_php is tough. A single upload can be handled with memory_limit (I wouldn't be too concerned if thumbnail creation fails for the occasional image), but that still allows for MaxClients (in the case of Apache httpd) * memory_limit MBs of memory in use by PHP. Any chance you can limit the number of connections (either by a FastCGI-like mechanism or by just offloading all static connections to another httpd) to a fairly low number?

author: ve7xen
author_rank: Super Contributor
timestamp: June 03, 2013, 05:56:54 pm
content: Quote from: gnif on June 03, 2013, 02:42:19 pmReally? I haven't used MySQL in anger in years (because it's a piece of crap next to Postgres), but is there not support for a lockless, consistent dump without taking the DB down? I thought MySQL had transaction support (via MVCC, like Postgres) these days. Or is this a problem caused by using MyISAM non-transactional, non-ACID storage backend?Anyway, even if that is a problem, you can still do the backup at a lower level. Hopefully you have LVM and can use LVM snapshots, which would work with a few seconds of downtime max. If not, you can do a hot copy with rsync, bring down the DB and then rsync again to make it consistent. Maybe a couple of minutes downtime if you do it that way.I'm still not clear on what the actual issue is if this problem will manifest even when backups are not being done. Is RAM over-committed? Are we hitting some MySQL/PHP/SMF bug? Or are we just talking about the probability of a case where too many long queries come in at once and the server gets overloaded to where it can't catch back up again?

author: gnif
author_rank: Unknown
timestamp: June 03, 2013, 10:33:16 pm
content: Quote from: alm on June 03, 2013, 05:04:56 pmWe already do limit the connections via the Nginx gateway, before we did so Google would take the site offline when it decided to crawl it. The problem with the thumbnail creation is that users on this forum like to upload high resolution images of their circuits of schematics, which when uncompressed for re-sampling consume 100MB+ of RAM. This is only an example of one use case that causes a memory spike.Quote from: ve7xen on June 03, 2013, 05:56:54 pmThis was not about how it is slow or bad to do this, Dave was asking if we could take a slow backup. Yes we could use LVM snapshots if the server was configured on LVM, but unfortunately it is not, and it is not likely to change as HG have configured this as a virtual server (KVM) on the dedicated hardware and wont budge on the decision. Even mysqlhotcopy has it's drawbacks, most notably it can not backup MyISAM tables consistently, which we still have a stack of.

author: ve7xen
author_rank: Super Contributor
timestamp: June 03, 2013, 10:55:39 pm
content: Quote from: gnif on June 03, 2013, 10:33:16 pmOh I know what Dave was talking about, but I don't think he'd be interested in a slow backup if the backups didn't have a huge impact on site usability/availability as they do now. I was just shocked that MySQL still can't do lockless consistent dumps, if that's in fact the case.Even without snapshots the DB files are only a few GB. I was suggesting that it might be preferable to take the DBMS offline for a short period (which you can minimize using rsync) to do a hot copy of the DB files rather than using a dump. I'm not sure how much downtime the existing backup process is causing; if you do it this way (rsync while live, take DB down, rsync again, bring DB back up) it should be under a minute, a few minutes maximum. That doesn't depend on any DB or OS features being available. And generates a planned, scheduled, and graceful outage; you can put up a 'backup in progress, please try again in 5 minutes' static page to serve users while it's down.

author: Mike Warren
author_rank: Unknown
timestamp: June 03, 2013, 11:16:26 pm
content: Quote from: EEVblog on June 03, 2013, 12:56:19 amThat's just crazy! Pop some RAM in your pocket, jump on a plane and tell them you want to admire your server. Then, while their backs are turned, pop in the RAM.

author: gnif
author_rank: Unknown
timestamp: June 03, 2013, 11:37:45 pm
content: Quote from: Mike Warren on June 03, 2013, 11:16:26 pmLol, the cost to fly over and do that would be more then the cost of paying for RAM for a couple of years. It costs that much because HG support it, if the RAM fails they take responsibility for it and replace it. I know that the cost is still a bit rich, but that is what they charge. I can understand it also because they are just reselling SoftLayer, and they charge the same.

author: xrunner
author_rank: Super Contributor
timestamp: June 03, 2013, 11:45:51 pm
content: Quote from: gnif on June 03, 2013, 10:33:16 pmHello gnif,Isn't the high resolution thumbnail images then one item that you need to apply limits to? I know Dave will not like it, but as you said, we're talking about something here or there that has to give way.

author: gnif
author_rank: Unknown
timestamp: June 03, 2013, 11:48:01 pm
content: Quote from: xrunner on June 03, 2013, 11:45:51 pmThis was one example, there are many other things that cause PHP's memory usage to spike... fixing this one thing will not resolve the problem.

author: xrunner
author_rank: Super Contributor
timestamp: June 03, 2013, 11:52:08 pm
content: Quote from: gnif on June 03, 2013, 11:48:01 pmYes, yes, I understand, but one step at a time. You mentioned it because it was having a detrimental effect. OK. How many more things would need to be limited in scope before it would begin to have an effect on the issue you are having?

author: gnif
author_rank: Unknown
timestamp: June 03, 2013, 11:56:59 pm
content: Quote from: xrunner on June 03, 2013, 11:52:08 pmThe other issues such as the inefficient queries would require major modifications to SMF, it is not viable.

author: xrunner
author_rank: Super Contributor
timestamp: June 04, 2013, 12:08:05 am
content: Quote from: gnif on June 03, 2013, 11:56:59 pmFirst, I will freely admit you are far more knowledgeable about the server part than I am. Case closed there. You win.But, I have been using SMF for many, many years, and I've been a member of the SMF support forum since 2007, and read thousands upon thousands upon thousands of support topics and I've never seen anyone having to muck about as much as you are having to do here. I've installed and run SMF forums for at least that long, and one I help with is bigger than this one as far as posts go.This forum reallyisn'tthat big as far as SMF forums go. It's a moderate size, and growing, but it really isn't that big yet (IMHO). Shoot it's barely bigger than my forum. From my perspective, the amount of twiddling you are having to do is a sign of some other problem. Put it this way - if it was this hard to run a moderate size forum like this one, hardly anyone would be doing it. And I can assure you, many, many people are doing it with few issues.Now, I know you're going to hit me with all manner of tech-savvy reasons that it can't work, but really, I still think something's amiss since, as I said, if it was this fucking hard, forums this size would be few and far between (and they certainly aren't that).

author: gnif
author_rank: Unknown
timestamp: June 04, 2013, 12:13:35 am
content: Quote from: xrunner on June 04, 2013, 12:08:05 amHere is the reason it is easy for you and hard here... your server's have more RAM. I have tried to explain the problem as simply as I can, and I am not the only person here to say that 4GB is pretty low for this kind of workload. Normally when a server starts to get slow because of RAM issues, the solution is simply to throw in some more RAM... most shared servers that are being resold are commissioned with 16GB of ram or more (in fact I did one just recently that had 48GB of ram).As an end user you never get to see how much effort goes into keeping these systems running smooth, nor should you. The difference here is that I can not just install more RAM as Dave has a limited budget... if I could, this thread would have never been created in the first place.

author: xrunner
author_rank: Super Contributor
timestamp: June 04, 2013, 12:18:46 am
content: Quote from: gnif on June 04, 2013, 12:13:35 amOK, our servers have more RAM. Not only my forum, but the other one I help with that has over 500,000 posts and is very active. What am I paying for the hosting account for my forum? The bare minimum. Peanuts. Yet I have more RAM available than you guys do?

author: Mike Warren
author_rank: Unknown
timestamp: June 04, 2013, 12:20:48 am
content: Quote from: gnif on June 03, 2013, 11:37:45 pmYes. I was just joking.But now a serious question, if you don't mind. (I know nothing about this stuff, so please excuse my ignorance): if shared servers have more resources for this sort of thing, why not stick with a shared environment? There are several hosting companies out there that allow unlimited bandwidth and storage for only a few dollars a month. Is it reallythatmuch slower?

author: gnif
author_rank: Unknown
timestamp: June 04, 2013, 12:40:15 am
content: Quote from: xrunner on June 04, 2013, 12:18:46 amThis is not my decision. You need to understand that I am assisting Dave with what he has, he was on a dedicated server, and was moving servers, and was plagued with issues since the move, so I offered my services. Since then we have worked out and fixed 99% of the problems and got the site running extremely fast.Here are reasons for not going to a shared environment.1) If another user on the server has a bad script/site then every user suffers, all sites run like crap until the admin finds & fixes the problem.2) If the server you are put on is oversold, you will also experience performance issues, and even complete outages.3) If a site on the server is hacked because a user cant be stuffed updating it, or can't, dave's site can be defaced/abused from the attacked site (no suPHP does NOT stop this, it just makes it harder, google 'apache symlink attack' for an example of why).4) On a shared environment, fastCGI or php-fpm is not feasable due to the security issues associated with it, and as such op-code cachers like APC can not be used.5) Due to the security requirements, memcached also can not be used.6) I know that Dave wants to be able to upload videos direct from the server using YouTube's API (I am still yet to do this), which can not be done unless you have shell level access to the server and permission from the admin to run such a high bandwidth utilisation script.7) Admins do not take kindly to peak traffic loads on their servers when you get published on a site such as HaD or Slashdot which I doubt your forum experiences.Ever wondered why this site is so fast compared with others? This is why.Here are the reasons for using a shared environment1) You don't have to give a rats ass about what the server is running, as long as it works you don't care.2) It is cheaper.This decision falls squarely on Dave, not me. At some point he decided to move away from a shared environment, perhaps he would care to comment why.Edit: It should also be noted that with additional ram I can also lift some of the Nginx limits on requests per second per IP as the server will be able to cope with more, making the website load even faster and handle peak load periods better.

author: xrunner
author_rank: Super Contributor
timestamp: June 04, 2013, 12:54:38 am
content: Quote from: gnif on June 04, 2013, 12:40:15 amI don't mean to be unappreciative gnif. Let's all remain friends right?We're just having a down-to-Earth discussion.QuoteOK, but I know of other large forums on shared servers that run just fine.QuoteI don't perceive it asthatmuch faster than other forums I visit. I just visited a few when waiting for you to reply. If you are talking in terms of tens or hundreds of mS, then does it really matter as far as a human brain is concerned?QuoteHave you ever seen the largest SMF forum? As far as I know it's this one -Warrior CatsNot my cup of tea, but they have 44.6Millionposts. Yes I said44.6 Million...Do they have a dedicated server? Yes, but I've read the person's posts on the SMF support forum over the years who Administers it and I've never heard of any of the twiddling you are trying to do with this forum. Never. If SMF was so bad, it wouldn't be able to cope with a 44.6 Million post forum as active as that one. This one is child's play compared to that one.So we're back to my burning question - why is this forum so hard to run and back up reliably? I'm very impressed with your knowledge yet ... something is not adding up for me. I can't speak for anyone else.

author: gnif
author_rank: Unknown
timestamp: June 04, 2013, 01:04:48 am
content: Quote from: xrunner on June 04, 2013, 12:54:38 amYeah, I am not taking any of this to heart either, who knows, I may have missed something your comments may make me realise, I am one that is ready to admit when I am wrong.Quote from: xrunner on June 04, 2013, 12:54:38 amI am in Australia, and when visiting any site outside of the AU I notice an enormous speed difference as there is around 250ms of latency in the round trip, adding another 100ms to the request brings that to 350ms for just the HTML... then add images, javascript, css and you will be getting to a few seconds to load the page. If you are running a business in AU for AU clients, you never get an international server even though we pay a premium for our bandwidth here, simply due to this perceivable lag.The lowest latency I have ever seen for an international server is 130ms to Singapore, but the link speed is terrible, Low latency but low throughput also.Quote from: xrunner on June 04, 2013, 12:54:38 amHow much RAM do they have? The twiddling is simply because we simply do not have enough and I have been trying to get this thing to run a good as it can with what we have. I promise you they will have more then 4GB.Edit: Just to be clear, I am not bashing on SMF in any way, I just see some design flaws that perhaps the developer did not realise could be done better/more efficient. I have been guilty of this on may occasions. The fact that SMF still uses the now deprecated mysql extension is a problem in itself and I will be submitting my patch upstream for inclusion in the project.Btw: I am not just a server admin, I used to be a full time professional developer, I am not talking as an admin who thinks he knows about software development, I am talking as an admin and a professional software developer. I have written entire shopping systems, forums, ticket systems, and help-desks in PHP from scratch utilising pretty much every technology under the sun (MySQL, MongoDB, SQLite, Beanstalk, Sphinx, MS-SQL via ODBC). These were for enterprise level systems consisting of hundreds of thousands of users in mission critical applications. I am fully aware of the issues with PHP, memory usage, MySQL performance and query optimisation.If you would like to see an example of my ability in the field of development please have a look atXBMC's AudioEnginewhich I wrote the majority of in my spare time for fun.Quote

author: xrunner
author_rank: Super Contributor
timestamp: June 04, 2013, 01:20:11 am
content: Quote from: gnif on June 04, 2013, 01:04:48 amCool.QuoteI don't know how much RAM they have. But if we back-engineer it , so to speak, we can assume that given enough RAM then SMF will be able to handle a forum that has 44.6 Million posts without many complaints.But, we're not dealing with that situation here. This forum is barely larger than mine, which, of you want to know, is this one -Atheist Think TankIt's on a shared server with a bargain-base hosting plan.Yet this one -Why Won't God Heal Amputeesis over twice as large as mine, and is on thesameshared server as mine, with the same plan (I don't own that one).So, I don't know man, I can't speak from your perspective, but I can speak from mine, which is years and years of forum experience and some with larger forums than this one. I think maybe you guys need todowngradeto a shared server, as crazy as that sounds.Peace.

author: gnif
author_rank: Unknown
timestamp: June 04, 2013, 01:27:50 am
content: Quote from: xrunner on June 04, 2013, 01:20:11 amOk, think of it this way..Linux Kernel needs say 10MB for itself,Memcached is taking 128MBAPC is taking 128MBMySQL is taking up to 2.1GBEach PHP process is taking as of right now, 447MB per process, but most of this is shared, actual usage is around 80MB per process.Lets say we have 10 visits occur at the same time..10 * 80 = 800MB.Do the math...10 + 128 + 128 + 2150.4 + 800 = 3.2GB with only 10 concurrent visitors.Then you also have to add the additional services on the server, such as cPanel processes, mail processes (dovecot, exim, etc).It is not about the size of the database. Even if we only got 20 hits a day, if they came in all at the same time, we would be out of RAM.Currently we are handing about 7-8 concurrent PHP hits at any one time, but when we get published on sites, or everyone signs on at once because it is COB Friday this increases.For those that are interested, we average a total of 60+ concurrent hits of all file types at any time.Quote from: xrunner on June 04, 2013, 01:20:11 amMaybe, but not my decision to make.

author: xrunner
author_rank: Super Contributor
timestamp: June 04, 2013, 01:53:40 am
content: Quote from: gnif on June 04, 2013, 01:27:50 amgnif,I've never, since I've been a member of the SMF support forum, which is here if anyone wants to research this issue -SMF Support ForumI've never read of anyone that was complaining about not enough RAM on their server for such a medium size forum as this is. Does that mean nobody ever did? No. But I've never, since 2007, ever heard of that complaint. Mostly it's how do I center the banner I made, or how do I stop spammers, etc. SMF seems to run very well for both large and small forums on shared servers, from what I've seen.QuoteBut, you never answered my question from another post. Are you a member of the SMF support forum? Why don't you ask these questions there? They are the experts, and theyDOhave server admins that know what the hell they are talking about.It's free, so what have you got to lose?

author: gnif
author_rank: Unknown
timestamp: June 04, 2013, 02:14:58 am
content: Quote from: xrunner on June 04, 2013, 01:53:40 amYes SMF does run well... but I reiterate... we have LOW memory for any PHP + MySQL application on a site this busy. It is not about the quantity of data in this instance, it is about how many requests per second we are getting that has to be dynamically rendered. 4GB is low for any combined PHP & MySQL server of any workload, it is the bare minimum and it is why it is the cheapest offering for dedicated servers.Quote from: xrunner on June 04, 2013, 01:53:40 amMy time and theirs trying to go through all this again when the problem has been 100% positively identified. I will say again, I have poured over the SMF code, I have analysed how it operates, and there is much room for improvement in its design even though it is a good PHP based forum.If they were as expert as you state the are they would be using an object oriented design, lazy class and database loading, taking more advantage of memcached, and seeing what MySQL reports about the queries they are writing by running them with 'EXPLAIN' (Note: these may be coming from 3rd party plugins, I have not searched for them in the code base). I will grant you that the forum is fast and works well, but from a software design perspective it leaves much to be desired for, and it could be much faster.You should know that the majority of PHP developers in the world are self taught based on tutorials on the internet, many of which do not go into anything beyond basic MySQL access, posting forms, emailing, etc. If they did we would not see so much crap PHP code around, nor would we see so many sites defaced because the programmer either did not know, or could not be bothered to properly secure his code. I was one of these once... and now I look back and I am horrified at what I wrote, going as far as to re-write entire applications to fix these design flaws at my own cost.Here are the top flaws I usually see in third party code, even in the commercial closed source world:1) Use of the deprecated mysql_* methods, either move to mysqli_* or even better, start using PDO. (The mysql_* methods will be gone soon completely, why continue to use them when they will disappear and break your application)?2) Failure to sanitize input strings properly that are used in queries or HTML output3) Failure to hash database passwords with a salt, or at all.4) Duplicating code over and over instead of just wrapping it into a class that can be reused across the application5) Not caring about how well the database performs and just assuming their queries will always be fast (eg: SELECT * FROM TABLE WHERE a IS LIKE '%b%')6) Writing SQL queries that span a few pages because the developer either did not know PHP well enough, or could not be bothered to implement the complex logic in the application.7) This one is a big one... website templates/plugins that allow auto thumbnail resize and accept the file-name in the query string. Often these scripts do not check the path they are passed, which means you can do this: "resize.php?src=http://my.site.com/my_malicious_file.gif".SMF is only guilty of 1, and I am sure this is just because it is legacy code that has not been updated in quite some time. I am quite impressed with SMF, but there is much room for improvement and I may take it on as a future project.

author: xrunner
author_rank: Super Contributor
timestamp: June 04, 2013, 02:35:00 am
content: Well gnif, why don't you guys switch the forum to a different one, such as phpBB, Vbulletin, or others?I reiterate, nobody I've ever seen complains about SMF performance as much as I've seen you complain about it. If you are the Bees Knees about it all, then why are you using SMF?

author: gnif
author_rank: Unknown
timestamp: June 04, 2013, 02:52:57 am
content: Quote from: xrunner on June 04, 2013, 02:35:00 amYou missed the point... I said I like SMF and I am quite impressed with it, but it could be better. Legacy code is to be expected in a project like this and it takes someone like me to get annoyed enough about it to fix it and provide the developers with the fixes before it gets fixed. Low priority on their radar, but high on mine. I am not complaining about SMF, I am noting that it has areas that could be improved which would benefit this particular site and configuration.If I was to dive into the code to fix things, the first thing I would fix is this query:Code:SELECTlo.id_member, lo.log_time, mem.real_name, mem.member_name, mem.show_online,mg.online_color, mg.id_group, mg.group_nameFROM smf_log_online AS loLEFT JOIN smf_members AS mem ON (mem.id_member = lo.id_member)LEFT JOIN smf_membergroups AS mg ON (mg.id_group = CASE WHEN mem.id_group = 0 THEN mem.id_post_group ELSE mem.id_group END)WHERE INSTR(lo.url, 's:5:\"board\";i:4;') > 0 OR lo.session = 'ipREM.OVE.ED.IP';MySQL itself is logging this as a bad non indexed slow query, it is not my opinion that it is bad, it is the databases. This query returned 79 records and had to search 1200 to do so. It could possibly be written either as two separate queries, or the database schema could be improved to prevent storing serialized data in the 'url' field of smf_log_online so that it does not have to perform a string search on every single record.I am not just saying it is slow because I feel like it, or I do not like it... I am saying it can be improved based on actual evidence and metrics provided by MySQL itself.

author: gnif
author_rank: Unknown
timestamp: June 04, 2013, 02:57:40 am
content: Here is another logged query that MySQL is reporting is slow... this one has to examine 250,000 records to perform this insert and takes 1.8 seconds.Code:INSERT IGNORE INTO smf_log_search_results(id_search, relevance, id_topic, id_msg, num_matches)SELECT233,1000 * (30 * COUNT(*) / (MAX(t.num_replies) + 1) + 25 * CASE WHEN MAX(m.id_msg) < 167946 THEN 0 ELSE (MAX(m.id_msg) - 167946) / 71978 END +t.id_topic,MAX(m.id_msg) AS id_msg,COUNT(*) AS num_matchesFROM smf_topics AS tINNER JOIN smf_messages AS m ON (m.id_topic = t.id_topic)WHERE (m.body LIKE '%brymen%' AND m.body LIKE '%rebadge%')AND m.id_board IN (2, 6, 21, 9, 11, 20, 22, 1, 3, 5, 7, 8, 4, 14, 19, 18, 16, 15, 13, 17)GROUP BY t.id_topicORDER BY nullLIMIT 1200;This could be improved by using an engine such as sphinx which someone previously suggested. It would have to be optional though as not all servers will have access to Sphinx.

author: xrunner
author_rank: Super Contributor
timestamp: June 04, 2013, 02:58:09 am
content: Quote from: gnif on June 04, 2013, 02:52:57 amgnif,This may sound sarcastic, but I really mean it. Put your money where your mouth is.Why not tackle this with the Big Boys at SMF, if you are so smart. I'm a nobody. I'm just a poor slob on the internet who is building a model railroad, and who has run SMF forums for many years. WTF do I know, right?Why not show them (the SMF team) how to do it right? Why not share your immense knowledge with the world there? Why not make SMF better? Anyone can submit bug reports and suggestions. If you aren't doing that, then why are you wasting all your knowledge on us - the poor uninformed masses?What is it going to cost you? The membership is free. What are you afraid of?I'll be waiting for a member named "gnif" to register, and I'll follow your every post there. I truly hopw you pass on your improvements to the SMF team, as they certainly can use them.

author: gnif
author_rank: Unknown
timestamp: June 04, 2013, 03:00:52 am
content: Quote from: xrunner on June 04, 2013, 02:58:09 amYou should read more carefully.Quote from: gnifQuote from: gnif

author: xrunner
author_rank: Super Contributor
timestamp: June 04, 2013, 03:04:24 am
content: Quote from: gnif on June 04, 2013, 03:00:52 amQuote from: gnifAnd you should act more decisively. Way more decisively.So, when will you join SMF and tell them "much room for improvement" and give them your suggestions?I'm waiting for an answer to that question gnif.

author: gnif
author_rank: Unknown
timestamp: June 04, 2013, 03:14:56 am
content: Quote from: xrunner on June 04, 2013, 03:04:24 amOk, this is just stupid, you are now personally attacking me and my ability, you have no knowledge in this area, just because you run a few SMF sites does not mean you have any idea what you are talking about, nor how the development process works in the open source community. I am not trying to say I am better then these other developers, nor am I trying to say that my way is the only way to go... SMF is full of legacy code that will be updated by developers at some point or another, how else do you think this community works?.I will be posting code to SMF, not by their community forums, but via their git repository ongithub. I let my code talk for itself, there is no need to tell other PHP developers how it is better or try to rub their mistakes in their faces, just provide the code in the form of a pull request and let them decide if they will take it or not.Providing a few patches to fix a few problems is NOT taking it on as a project... I stated that I WILL be providing this MySQLi patch upstream for inclusion, how more decisive do you want me to get?

author: GeoffS
author_rank: Unknown
timestamp: June 04, 2013, 03:28:27 am
content: This thread is getting way off topic. It was intended as a way for users to report a 504 event and for the administrators to report progress, not for users to tell the administrators what they are doing wrong.

author: EEVblog
author_rank: Unknown
timestamp: June 04, 2013, 04:47:09 am
content: Quote from: xrunner on June 04, 2013, 01:20:11 amI ran this forum for a year or two on a sub $10/month shared server. On top of that it was also running the wordpress blog, several other high bandwidth HTML websites of mine, AND hosting streaming podcast video!Yes, it worked, but it eventually got to the point that it was dog slow in fits and spurts, it became unpredictable. Often it worked like lightning and you'd think shared servers are great, but then it would take 20 seconds to post your message. No one really complained until the issue was actually raised, they just put up with it. Some I only found out later stopped using the forum or didn't start to begin with because it was ultimately so unpredictable.I then moved to double priced "cloud" based server which was a nightmare for all sorts of technical reasons.The best thing I ever did was move to this dedicated server. Yes, it's 20 times the cost of a cheap shared server, but I'm sure everyone will agree that the performance is lighting quick and completely predictable.Yes, I'm paying a lot to host the forum and website, yes I'm probably being ripped off and it can be ultimately be done a cheaper way. But I want a forum and website that works as fast as possible without fuss or interruption, and can handle a slashdot or whatever.