What would one say is the sweet spot for second hand servers? - Page 1
Forum section: electronics / general

author: Ampera
author_rank: Super Contributor
timestamp: February 15, 2020, 02:22:45 pm
content: I've decided to start looking for a new server, something I can use as a proper Xen hypervisor, with the potential for cores, computing power, memory, and reasonably fast networking. Now whenever I look for more than what I can afford, I start to look second hand, and then sorta get in an awful loop of never being sure if what I'm looking at is /truly/ the biggest kerbang for my buck.What I've been looking at primarily are quad socket Intel based systems, particularly Ivy Bridge. They are cheap on ebay (which means easy to find in my book), have reasonable performance, and 12 core E5's are not hard to come by for semi-reasonable prices. That being said, I need to know if I'm missing something, so I'm asking people here if they have any particular suggestions.I had the idea to look for Haswell gear, but it's particularly expensive at the moment, likely since it's still in considerable use, but my hope being that since there's so much of it, it will eventually crash in price as it all gets decommissioned. At the end of the day, it needs a lot of cores, and the potential to be easily upgraded, meaning I want to spend relatively small investments to get equal increases in performance, instead of having to put massive dollars on a new machine to replace it a few years later.The last comment I'll make is that someone (hi halcyon) will say "Server Auction Houses", but I don't know of any around where I live, or even ones that'd be willing to ship stuff. I live near Albany, NY, which isn't exactly a hotspot for datacenters. If someone would be kind enough to give me some examples, I'd be most grateful.

author: OwO
author_rank: Super Contributor
timestamp: February 15, 2020, 02:35:05 pm
content: If this is going to be on 24/7 electricity costs will be significant compared to the cost of the server, so be sure not to overlook this. Power management on servers are typically bad last time I checked, so I think you will save money in the long run by using a desktop PC. I don't see any point in buying old computer hardware because performance per watt is still improving, and hardware becomes obsolete fast. I upgrade as infrequently as possible (~5 years), but when I do I generally go for the latest top of the line stuff.

author: Ampera
author_rank: Super Contributor
timestamp: February 15, 2020, 03:07:37 pm
content: Quote from: OwO on February 15, 2020, 02:35:05 pmThe issue is that desktop hardware scales like crap. Think, if I can spend ~1200 USD and get a 4 CPU 48 core Ivy Bridge server, to get that same core count (remember, VMs, individual core performance is not as important if I can't divide them up as well) I'd need to have multiple systems, which means multiple PSUs, multiple motherboards, multiple NICs, and a lot of complicated networking gear to get them to work together. That also means multiple Xen hypervisors, and a /lot/ of overhead, diminishing the power/performance even harder.I'd definitely love to purchase all brand new hardware, but it is expensive, beyond what I'd be willing to pay. Believe me when I say energy is /not/ a concern, and a difference seen on the order of a few years would have to be a lot to make me reconsider.When I buy second hand servers, keep in mind I'm buying an entire system too, whereas if I buy new hardware, I'm pretty much stuck buying individual parts, which rack up quickly.

author: RoGeorge
author_rank: Super Contributor
timestamp: February 15, 2020, 03:11:11 pm
content: Do you have any idea how noisy a server is? It's unbearable unless you dedicate a room for it, or a small closet, but then you'll need climatisation and so on. Also,why do you think you need Xen? Unless this is for a medium company, it doesn't make sense for home use.Went myself to the same path, Xen, then KVM, then realized that an Ubuntu LTS with ZFS for daily use is more than enough, requires zero maintenance, and can still do virtualization. Another separate disk, which normally stays unpowered, is for gaming only (dual boot by physically switching the main and the gaming disks, so they are never powered at the same time).Virtualization is very cumbersome to setup, both in hardware and in software, video passthrough is almost impossible to set and maintain, and the whole mess is NOT hack-proof, anyway, if that is what you want to achieve. Nothing is hack proof.

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 15, 2020, 03:25:24 pm
content: Quote from: RoGeorge on February 15, 2020, 03:11:11 pmYes yes yes. They are noisy as hell and are electricity whores.Unless you really need to serve a large number of connections/users 24/7, I'd go for good desktop systems instead. I realize that oldish servers can be had for relatively cheap, but you'd probably regret it.

author: OwO
author_rank: Super Contributor
timestamp: February 15, 2020, 03:49:15 pm
content: What do you need a 48 core server for? If this is a hosting operation, do you have a 10Gb/s internet feed to go along with it? I would first look into whether a massive server is really needed or if a 8-core desktop PC can be sufficient. If you simply need a lot of isolated zones, I think it's a better idea for them to be physically separate machines if security is important.

author: SilverSolder
author_rank: Super Contributor
timestamp: February 15, 2020, 04:09:17 pm
content: Ask yourself two questions:1) What workload do you have...2) Does the load scale across multiple cores?You may be better off overall with two 12 core machines than one 24 core one - for both redundancy and performance. Likely to be simpler/cheaper too. And easy to scale to as many cores as you could ever dream of!

author: olkipukki
author_rank: Frequent Contributor
timestamp: February 15, 2020, 04:59:01 pm
content: Quote from: Ampera on February 15, 2020, 02:22:45 pmIf you are after old Intel's garbage, E7 would be more or less in same price category (with patience and ebay fishing),but give you extra few cores and probably performance boost.Quote from: Ampera on February 15, 2020, 02:22:45 pmThere must be a good reason why put all your eggs in one basket...

author: Mr. Scram
author_rank: Super Contributor
timestamp: February 15, 2020, 05:23:04 pm
content: You have to understand these old servers are sold exactly because they've become too cumbersome to run. And because their support contracts ran out I suppose, but the point is that people felt they were better off buying new ones. Ivy Bridge is at this point old hat of nearly a decade old.

author: Ampera
author_rank: Super Contributor
timestamp: February 15, 2020, 05:42:03 pm
content: Quote from: RoGeorge on February 15, 2020, 03:11:11 pmYes, I do know how noisy servers are. I have some, they take energy, put out noise, and do things. I still want a good one.Quote from: RoGeorge on February 15, 2020, 03:11:11 pmAs for Xen, I have a good idea of what I'm getting myself into, and I have a long and successful history of maintaining complicated and intricate systems. My current solution is a Xeon X5650 running FreeBSD , but I have outgrown this considerably. I'm also going to be sharing this server with other people, who will be running their own software and operating systems, which is why I want Xen.Quote from: OwO on February 15, 2020, 03:49:15 pmThis is almost a hosting operation, but within a couple or few people. In general, I like powerful hardware, and I have genuine application for it. I run a lot of my own services because, and this is one of the considerable reasons behind me purchasing a large server, /I want to/I didn't fall off the turnip truck yesterday and start having wet dreams about some mystical contraption known as a server. I've dealt with servers for years, and I enjoy doing it. It's my hobby, and expanding upon that for me is educational and entertaining.As for internet, I have a symmetrical gigabit connection, not that this is too important since all the server users will have near constant physical access to the machine, but being the point of a server is to, well, serve, I have a reasonable connection to do that on.Quote from: SilverSolder on February 15, 2020, 04:09:17 pm1. A large variety of applications varying from web, email, and file, to virtual workstation, game server, and experiment hosting.2. Yes.As for the core count, I've had my eye on a 4 CPU 6 core Xeon machine that's going for around 600 bucks on ebay, comes with 128GB of RAM in two nodes. I'm not a fan of having two nodes, which is why I'm still looking, but the example serves to illustrate a point I am trying to make. I am trying to spend money in increments, such that one machine can be expanded into faster and faster solutions.Quote from: Mr. Scram on February 15, 2020, 05:23:04 pmI understand that, but for my purposes (and more importantly, budget), they still provide me with the performance I require at a power consumption I'm not upset by. It's also why I do want to edge closer towards newer equipment. My current X5650 is actually quite good, and is yet older than Ivy Bridge, so I have some absolute bottom line of performance expectations with it.Now, as for a few other questions people are asking:1. Why don't you use desktop hardwareA: Because it doesn't scale well, the power consumption will be higher for the same performance range, and I genuinely believe I need, and do want a high core count server, designed to be a server.2. Why don't you just use multiple machines?A: Because I already am. This is intended to be one server out of 3, with the other two being X5650 storage servers holding SATA SSDs and/or hard drives.The one and only good suggestion I've seen is to look for E7 in particular. This crossed my mind, but I had the thought that E7 might end up being significantly more expensive than E5s. I'll keep looking around with that in mind.tl;dr I know what I'm doing, I'm not stupid, I've taken all of your concerns into account before you've placed them, and I am perfectly willing and able to maintain a Xen environment. While I understand the wish to ensure that people don't do stupid things with money, I'd kindly request that people let me worry about that one. Not trying to be rude or dismissive, though it may seem.

author: mariush
author_rank: Super Contributor
timestamp: February 15, 2020, 05:55:29 pm
content: Considering how much performance those Intel processors lost due to vulnerability patches and fixes (and more fixes and patches are coming) buying old Intel servers is less and less tempting.Unless you need some of those features servers have like remote administration, you would probably be better off with Ryzen processors.There's Ryzen 1600 AF which is basically a rebranded Ryzen 2600, for around 100$. It's a 65w TDP cpu and you can combine it with a 60-70$ motherboard ( you could even get a cheap A320 based motherboard but you're better off with a B450 based board) and some DDR4 ram and you get 6 cores / 12 threads and very good performance, probably better than what those old Intel server CPUs offer.

author: SilverSolder
author_rank: Super Contributor
timestamp: February 15, 2020, 06:00:56 pm
content: Running everything on one box is not best IT practice - you end up having to kill everything if the box has to be restarted for whatever reason (and they always do... and the more stuff you do on the box, the more likely/frequent it becomes).I would build a very fast machine for GUI/gaming etc, and have a couple of lighter servers as well for stuff that needs to be "permanently on". You will be happier in the long run.

author: Ampera
author_rank: Super Contributor
timestamp: February 15, 2020, 06:02:20 pm
content: Quote from: mariush on February 15, 2020, 05:55:29 pmWhich I'm not throwing out of the realm of possibility. Nothing is decided until the money is spent, but I need to be able to scale that up. I did think about Meltdown and friend, which is why AMD CPUs might not be as nasty as they once were, but I've still heard bad things about them dependability wise.

author: Mr. Scram
author_rank: Super Contributor
timestamp: February 15, 2020, 10:46:04 pm
content: Quote from: Ampera on February 15, 2020, 06:02:20 pmForget what you know about AMD and especially in relation to servers. AMD currently has a competitive and many would say superior offer even before cost is factored in. A mundane modern AMD desktop will beat the pants off of most decade old servers and more workstation or server oriented hardware is ridiculous even compared to relatively recent hardware of a similar pedigree. The enterprise world seems to recognize this as sales are booming. It also means that older hardware has become relatively uninteresting as it's left in the dust.

author: Johnny10
author_rank: Frequent Contributor
timestamp: February 16, 2020, 12:21:28 am
content: Wouldn't a couple of Dell R720 Servers do for your needs?E5-26xx v2 compatibleAll the server bells and whistles for next to no money.If you wait for an Auction I see them go for 140 with no hard drives.If you want them in particular configuration you pay more.There are 35000 hits on eBay.Why people are buying the promoted ones?? I imagine because they are still quite viable.Looking through sold listshttps://www.ebay.com/itm/Open-Box-Dell-PowerEdge-R720-Win-Server-2008-4x-600GB-HD-Intel-E5-2630-0-4GB-RAM/254490834671?epid=28028154196&hash=item3b40d5f6ef%3Ag%3Awk0AAOSwgzZeEzi3&LH_Auction=1

author: tszaboo
author_rank: Super Contributor
timestamp: February 16, 2020, 01:14:12 am
content: I have 2x10 cores in the garage, with 128GB RAM. Supermicro. Go with 8 or 10 cores, the 12 core one is slow on GHz.I can tell you, forget all this proprietary hypervisors. Install Proxmox on it, and thats it, done. All you need is a flash drive, monitor keyboard while installing (not necessary if you use the proprietary network interfaces and pay more for software licenses than the server) and a web browser.HP is a plague, avoid. They want money for firmware update on a second hand server. Dell is OK, but I ended up with this, as it was cheaper than Dells. And most configs forgot to mention, they come without HDD racks. There is a huge lack of Racks, because companies like Facebook and Google destroy it together with the drives, so they could be 15 EUR each. If you have 12 of them in your server, that is costly.And go with 2.5inch, unless you want to hoard data, like me.

author: Red Squirrel
author_rank: Super Contributor
timestamp: February 16, 2020, 06:03:13 am
content: The biggest issue I find with used servers is the shipping cost, and often exchange rate. Sometimes customs. By the time you factor all that in, you're probably better off building a box with more modern hardware, even if you go with desktop parts. I kind of want to build a Ryzen based VM cluster and run Proxmox so I can get off ESXi (single host right now) and be able to do HA, live migrations etc. When I was checking out parts it seems you can get motherboards that support 64GB of ram on that platform. 3 Ryzen based machines with maxed out ram would make a respectable home cluster. For home/small business use there is not a huge advantage going with server grade other than the extra cost and extra power usage. Rack mount cases that are half decent are hard to find though, I'll just build my own for my next builds I think.

author: OwO
author_rank: Super Contributor
timestamp: February 16, 2020, 08:42:05 am
content: I highly recommend not mixing network services (web, file, mail, etc) with local terminals (VMs serving thin clients) or (!!!) local users. I thought the server and thin client model was obsolete since the 18th century.What I would do: get a separate workstation for each user. Have network services and file server on a dedicated, low power server.You keep saying you want the server to "scale", but the whole takeaway from the last few decades of datacenter development is that single serversdon'tscale. Inherently you can not have the memory locality necessary to keep the illusion of a single server once you go beyond one CPU package, so any server with more than one CPU is using a NUMA (non uniform memory access) architecture, which means each VM you run really need to be locked to a single NUMA zone (= CPU) with dedicated memory allocation (from local zone only) to perform well, which defeats the point of a "single server". For example if you have a 2-zone server with 32GiB of memory on each zone, you can not run 3 20GiB VMs and have it perform well, because one VM will be split between the two zones.In general VMs don't talk to each other except over the network, so separate physical servers will look the same. You can get very small form factor motherboards these days with very low overhead power consumption, and it doesn't take much effort to design a power distribution network to power everything from a single telecoms PSU.

author: SilverSolder
author_rank: Super Contributor
timestamp: February 16, 2020, 12:58:57 pm
content: Quote from: NANDBlog on February 16, 2020, 01:14:12 am+1 for Supermicro, I've used their boards for years. Great bang for the buck and high quality stuff.

author: olkipukki
author_rank: Frequent Contributor
timestamp: February 16, 2020, 09:23:29 pm
content: Quote from: Ampera on February 15, 2020, 05:42:03 pmWhat exactly are planning to expand? Memory? Replace CPUs?Beefy E7 mostly used in 4U storages and if you keep existing external storage, you basically buying a metal scrap.As example,https://www.ebay.com/itm/Supermicro-24-Bay-4U-X10QBi-4x-E7-4860-V2-Rails/173923363287upgrade sometimes withhttps://www.ebay.com/itm/Intel-XEON-E7-8880-V2-SR1GH-15C-2-5GHZ-37-5MB-FCLGA2011-CPU-Processor/202884474554Quote from: Ampera on February 15, 2020, 05:42:03 pmHope that at least you checked some numbers that 'a new server' will not fall too much behind your current X5650https://www.cpubenchmark.net/multi_cpu.html

author: Ampera
author_rank: Super Contributor
timestamp: February 17, 2020, 01:28:55 am
content: Quote from: NANDBlog on February 16, 2020, 01:14:12 amI'm not sure how Xen is more proprietary than Proxmox. Xen is a completely free and open source type 1 hypervisor. I don't like using solutions like Proxmox because, since this is a hobby for me, I don't get anything out of putting some software on a box and calling it good. I want to put it together myself.Quote from: OwO on February 16, 2020, 08:42:05 amVirtual workstation stuff is intended to provide graphical environments multiple people can use at one time. It's a personal experiment, and this is NOT a production environment.Quote from: OwO on February 16, 2020, 08:42:05 amWhich is a fair point, and utilizing multiple, cheaper servers is not outside of the realm of possibility. When I said scale I meant I don't believe desktop hardware will have the ability to be upgraded as well as server hardware. In general my concern with having multiple servers is having to run multiple hypervisors, which makes it harder to manage, and will take more overhead than having one sever with multiple processors.Quote from: olkipukki on February 16, 2020, 09:23:29 pmI have no clue what you're trying to tell me here, my X5650 cost me around 70 bucks for the CPU, motherboard, and some RAM. Hard drives and additional memory was more. New server hardware is prohibitively expensive for me. While new desktop hardware might have the passmark numbers, they do so with considerably fewer cores, which makes it much much harder to split into multiple VMs, even if I put significant money down into something like Ryzen 9 or Threadripper.I think people need to understand I am not a business, I am not running a for-profit hosting operation, I am a couple of morons looking to have some fun with old hardware, and to learn something from it. I think people are taking me a bit too seriously, and I'm starting to regret asking what I thought was a simple question. What hardware has good price to performance in a hypervisor setting. Do also keep in mind that even though I might not be getting the answers I like, I appreciate everyone's time spent into replying in general.If I can be shown, with evidence, that for a hypervisor, new hardware will have significantly better performance WITHIN MY PRICE RANGE, with a number of VMs, I'll consider it, but as far as I can see, old hardware is where I should be looking.

author: SilverSolder
author_rank: Super Contributor
timestamp: February 17, 2020, 04:07:29 am
content: The old hardware idea is sound enough. Getting a quad socket server of some kind up and running sounds like fun, even if not necessarily entirely practical. When all is said and done, do something cool/fun and don't forget to post the pictures!

author: olkipukki
author_rank: Frequent Contributor
timestamp: February 17, 2020, 09:08:27 am
content: Quote from: Ampera on February 17, 2020, 01:28:55 amIt seems misunderstanding here. 'a new server' means your 'new' server, nothing else, whatever it will be finally E5 or E7 (or something else), passmark give a hint regarding expected performance. Hope you are buying not just for sake of quad CPU...Not sure how these related to Ryzen or Threadripper

author: Towger
author_rank: Super Contributor
timestamp: February 17, 2020, 10:13:01 am
content: Loud and noisy, big and heavy, use lots of electricity, need a very deep rack to keep them in. BTW don't forget you may need it's propriety rack railsOk, got that over with...You need contacts in the right kind of local companies. Typically they will buy servers with the full metal jacket service cover. 365 days, 24 hours and 3 years. After 2 years the server is scheduled for replacement, it will be decommissioned at 2.25 years, sit around for 6+ months and then skipped. The trick and hard part is to get it before it is skipped. Time it right and you may have a server with couple of months of 24H call out warranty left :-)

author: tszaboo
author_rank: Super Contributor
timestamp: February 17, 2020, 11:44:46 am
content: Quote from: Towger on February 17, 2020, 10:13:01 amSits in the garage. You place it once there, open it once or twice, maybe slite in a HDD. The rack is only needed if you rack mount it, but why would you? BTW, the LACKRACK is a thing:Quote from: Towger on February 17, 2020, 10:13:01 amThey typically go into a datacenter and then they thow them out after a while. You dont want a SOHO server second hand, thats going to be expensive. You want something that big company like Google/Amazon/Facebook used, because they flood the market with used parts. The pricing of used server hardware doesnt make sense, because something, they have 1000 on stock, with no buyer is going to be a lot cheaper, than some obscure part.And no, you dont get a contract. Spend 500-1000 EUR on a used part, and use it for a few years.OP:Also, this reddit thread for inspiration:https://www.reddit.com/r/homelab/

author: Ranayna
author_rank: Super Contributor
timestamp: February 17, 2020, 01:26:53 pm
content: Wow, I remember those 1950s and 2950. Loud beasts they were. I cannot imagine that those things are really effective nowadays, seeing those SAS bays puts them at about 14 years old...Compare that to the more recent 730 (also old now) 2U Rack servers. Those are actually quite silent when they are cold enough. Keep those in mind for an upgrade.Another, comparatively silent server is the DELL 2900, the tower version of the 2950. That one was also silent.

author: Towger
author_rank: Super Contributor
timestamp: February 17, 2020, 02:56:51 pm
content: Quote from: NANDBlog on February 17, 2020, 11:44:46 amThat looks like an Ikea table, you don't show what holds up the server's weight at the back :-) I think what Google/Amazon/Facebook use these days it totally custom built and not going appear on the second hand market, except maybe in the endless supply of on eBay etc of cheap server CPUs and memory from China.

author: tszaboo
author_rank: Super Contributor
timestamp: February 17, 2020, 04:52:40 pm
content: Quote from: Towger on February 17, 2020, 02:56:51 pmNot my picture. But why would you need to support it at the back? 19 Inch is designed to be mounted with bolts on the front.Here is a ebay listing for ex google search appliance:https://www.ebay.com/itm/Dell-Google-R710-SFF-GB-7007-Gen-II-12-Core-2-66GHz-X5650-24GB-H700-No-2-5-HDD/174172343667?epid=1801409820&hash=item288d7b0d73:g:7HsAAOSw8bteL0YyThey used various Dell R710s, which were not branded like that cheese above.Even though a company will use their own format for the servers, it will still use a lot off the self servers. Because your own server lead time might be, such that you miss a market. Then you just buy stuff from HP, or Dell, whatever you can get your hands on.

author: Ranayna
author_rank: Super Contributor
timestamp: February 17, 2020, 05:11:21 pm
content: *Any* Server, except maybe one of those crapply low powered appliance types, needs to be fixed at the back.Even a proper, stable and massive rack will not hold a PowerEdge 2U Rack server if there is nothing below. Those are just too heavy. The rack rails would bend, or maybe even the "ears" of the server could snap off.The IKEA table in the picture likely has some bracket mounted on the rear legs.

author: Monkeh
author_rank: Super Contributor
timestamp: February 17, 2020, 05:32:07 pm
content: Quote from: NANDBlog on February 17, 2020, 04:52:40 pmHave you ever installed a piece of rack mount equipment bigger than a switch?

author: Towger
author_rank: Super Contributor
timestamp: February 17, 2020, 09:46:41 pm
content: Without any support from the back, the weight of those servers would rip the wood screws from the table legs. They are over 2 feed long and can weight ~25kg each.Get you hands on an old one and take it apart, the build quality is much higher than a standard PC.The Google appliances were sold/leased to companies to index their internal documents. I don't know how much they differ from what they use internally. But, I believe the they have long since moved to blades, running from a common (data center) 48v volt bus.Years ago the likes of Yahoo used bog standard generic desktops. The idea was was they were much cheaper than a proper server.

author: tszaboo
author_rank: Super Contributor
timestamp: February 18, 2020, 09:25:03 am
content: Quote from: Towger on February 17, 2020, 09:46:41 pmYeah, probably. I see people placing L profiles on their lackrack.Quote from: Monkeh on February 17, 2020, 05:32:07 pmNope, biggest thing was rackmounting multimeters.

author: Haenk
author_rank: Super Contributor
timestamp: February 20, 2020, 06:50:36 am
content: I pretty much was in the same situation a couple of weeks ago - our storage was close to the limit and the disks reached their end of lifespan. So I went with a 24bay LFF used Supermicro, with Trays, 64GB of RAM, a very nice RAID-card with BBU plus all the trays, dual E5-2670, dual PSU - for just 750,- EUR shipped, including a 1-year warranty. That's a fracture the costs of a new server, you won't save that Money on power costs. (I might pull one CPU, as one CPU is enough for the job). 24x8TB, running on RAID6. Server is running on Windows10 (Desktop), that's good enough for a two-person use with lots of data.

author: Mr. Scram
author_rank: Super Contributor
timestamp: February 20, 2020, 09:04:34 am
content: Quote from: Haenk on February 20, 2020, 06:50:36 amAre you sure about the power consumption? Those are rather ancient servers by now and definitely not frugal. The E5-2670 boxes on offer often are old Facebook servers they got rid of a few years back. They flooded the market and were surprisingly cheap back then.

author: borjam
author_rank: Unknown
timestamp: February 20, 2020, 09:38:29 am
content: Quote from: OwO on February 16, 2020, 08:42:05 amHence ransomware, mainteinance hell...