DISCUSSION: Raspberry Pi Mass Cluster - Page 1
Forum section: electronics / projects

author: pbgben
author_rank: Contributor
timestamp: April 11, 2014, 02:30:57 am
content: With the recent release of the Raspberry Pi Compute Module I've been thinking of what new possibilities this could bring to the hosting market.So, What I had in ind was a discussion about how many of these modules you could get working and in what space, E.G. a 1U rack space.Ill start,I'm not very technical but you could easily fit 300+ in a 1u space. Im not sure how you would network them or power them in the same space though.

author: T0mn0mn0m
author_rank: Newbie
timestamp: April 11, 2014, 02:49:55 am
content: A cluster with the new SODIMM form factor raspberry pis would certianly be cool. You probably could fit 300+ in 1u of rack space if you had the funds. Correct me if I'm wrong but I think the new form factor rasppberry pis lack ethernet, or at least have the ethernet chip but lack the jack with the ethernet transformers. So networking them could be tricky, you would probably need a seperate switch and a whole bunch of ethernet transformer chips for that.Or if fast networking isn't required, I suppose they could be networked over GPIO with say 20 raspberry pis per mini cluster, and 5 networked with faster communication to a controller pi, or a little mini itx board or something.Apart from just for the fun of it I can't see any real application, and as 20 of these boards will set you back $600 it's an expensive project.

author: pbgben
author_rank: Contributor
timestamp: April 11, 2014, 02:59:19 am
content: Networking will be a massive issue in my eyes, Maybe someone with the know how can shed some light.As for the practicality, say we can fit 250 per U, and we use a 42U rack thats10,000RasPis with 2U to use for networking etc,10,000 Pis = to 10,000ghz CPU and 4.8tb RAM. all from $300,000 plus the hardware to link them together

author: GeoffS
author_rank: Unknown
timestamp: April 11, 2014, 03:56:28 am
content: The new compute module has no USB or ethernet ability. That's provided by the motherboard into which the compute module plugs.The motherboard shown in thevideoprovides USB and HDMI but not ethernet.

author: theatrus
author_rank: Frequent Contributor
timestamp: April 11, 2014, 05:00:58 am
content: The RPi SoC has native USB, but does not have native Ethernet (it is after all a mobile phone SoC). The main Rpi boards run an on-board USB<->Ethernet chip (http://www.microchip.com/wwwproducts/Devices.aspx?product=LAN9512)

author: EEVblog
author_rank: Unknown
timestamp: April 11, 2014, 05:26:52 am
content: Quote from: T0mn0mn0m on April 11, 2014, 02:49:55 amHow does the combined computer power of these 20 boards compare other existing compute solutions costing that same $600?

author: EEVblog
author_rank: Unknown
timestamp: April 11, 2014, 05:35:36 am
content: Quote from: EEVblog on April 11, 2014, 05:26:52 amThe answer seems to be that 20 barely equals a modern PC:http://raspberrywebserver.com/raspberrypicluster/raspberry-pi-cluster.html

author: theatrus
author_rank: Frequent Contributor
timestamp: April 11, 2014, 05:56:44 am
content: Quote from: EEVblog on April 11, 2014, 05:35:36 amIf the efficiency was good, a lot of serving is strictly I/O bound or wants lots of little failure domains (where managing 1M little servers is not a big deal). However, the RPi isn't going to excel in either area.

author: ejeffrey
author_rank: Super Contributor
timestamp: April 11, 2014, 06:34:36 am
content: Quote from: theatrus on April 11, 2014, 05:56:44 amExcept that when you plug these into a backplane with networking and a power supply you add a bunch of single points of failure that can take out dozens of 'independent' failure domains. Other than a gee-whiz factor, there isn't a lot to recommend this approach over a software based container / virtualization system.

author: theatrus
author_rank: Frequent Contributor
timestamp: April 11, 2014, 06:37:51 am
content: In theory, you've fully isolated software (OS/Kernel) faults, but I agree, this is just gee-whiz. Intel hardware ends up being amazingly efficient (performance/$, performance/watt, performance/m3).

author: Someone
author_rank: Super Contributor
timestamp: April 11, 2014, 09:53:17 am
content: Quote from: theatrus on April 11, 2014, 06:37:51 amperformance/watt usually gets topped by FPGA or DSP computing.

author: tszaboo
author_rank: Super Contributor
timestamp: April 11, 2014, 12:04:18 pm
content: There are already ARM srevers in the pipeline:http://armdevices.net/2011/11/06/arm-servers-getting-ready-to-disrupt-intels-50billionyear-server-market/I've seen news from HP, IMB about it. But I dont think the Pi would be really a cost effective product for this. The processing power is way too low for that. The same day the Pi was released at Embedded world 2012, I was holding a Tegra 2 development board in my hands, which did everything what the Pi did and more. To start with, it is based on the old ARM11 architecture, not the Cortex. It is more hyped than what it deserves IMHO.The new Pi computemodule also looks good on pictures. "Integrate the Pi in your system", good hype. But, as already written, id doesnt have ethernet, it doesnt have proper USB (at least 1 for debug, one for user, and one for hosting). The reason to chose this kind of module is to keep the costs down on the mainboard. You outsource the 6-8 layer BGA work, and you dont have to worry about it, it is tested, OS is ported to it. Integration time is low. That is the idea.Now, if you want ethernet and USB, you need a controller for each. And your mainboard is not cheap anymore.So it will be a hit and miss. Miss because of the lacking features, and a hit because it will be most likely cheaper than the Pi itself, and half the price than anything we already have. I hope it will remind the manufacturers for the need of the engineers, and at some point we get some "pi killer" usable product at a great price. Like an Alwinner module from Olimex.

author: gxti
author_rank: Frequent Contributor
timestamp: April 11, 2014, 12:44:32 pm
content: I agree that the most interesting thing about the "Compute" module is that it will encourage other manufacturers to do better. However, despite being an overall Pi-hater I still think it's a nifty board considering the price and I'd consider using it. It's not the first module in that niche, but to my knowledge it's the first at such a low price point.As for filling a rack with them, that's just silly. Let's say you can fit 600 of them in 2U as someone threw out earlier. That's $18,000 in modules alone, not counting the backplane and its supporting circuitry, redundant mains power supplies, USB-to-Ethernet and Ethernet switch ICs, management console of some sort (how do you multiplex 600 UARTs? FPGA probably). Plus you still have to cool it all.$18,000 can buy a seriously ballin' real server that fits in the same space and has all of that, including the 1TiB of RAM and enterprise SSDs. Add virtualization and you can run as many "nodes" as your heart desires.

author: Rigby
author_rank: Super Contributor
timestamp: April 11, 2014, 12:58:29 pm
content: Doesn't this question belong on SlashDot? A running joke comment there for a long time was "Imagine a Beowulf cluster of [very slow computing device]!"

author: pbgben
author_rank: Contributor
timestamp: April 12, 2014, 12:12:50 pm
content: Ok, so it may not be the best performance for your money, but I'm sure there is still a market for a "Clustered" setup.I've created a simple design concept.

author: pbgben
author_rank: Contributor
timestamp: June 21, 2014, 01:07:53 pm
content: Are their any PCB designers here that can lend me a hand? I'd like to get some prototypes done so I can start crowd funding.

author: sync
author_rank: Frequent Contributor
timestamp: June 21, 2014, 01:33:12 pm
content: Quote from: pbgben on April 12, 2014, 12:12:50 pmWhich markets?

author: pbgben
author_rank: Contributor
timestamp: June 21, 2014, 01:43:38 pm
content: Quote from: sync on June 21, 2014, 01:33:12 pmEducational and Hobby users mainly, But this system can be used buy anyone that is in need of a low power cluster.

author: sync
author_rank: Frequent Contributor
timestamp: June 21, 2014, 01:57:08 pm
content: Quote from: pbgben on June 21, 2014, 01:43:38 pmHow much hobby users will spend >1000 bucks for such a toy?And professionals won't touch this. For example no one sane uses 100s of GB non-ECC RAM. Too much hassle.

author: pbgben
author_rank: Contributor
timestamp: June 21, 2014, 02:23:58 pm
content: Quote from: sync on June 21, 2014, 01:57:08 pmNot all hobbyists will be able to afford one for them self, but a group of them could easily afford one. As for ECC RAM, I don't see it being much of an issue.

author: tom66
author_rank: Super Contributor
timestamp: June 21, 2014, 02:33:36 pm
content: You have never designed a PCB before but you expect to be able to manage a crowdfunding campaign? This will be another Mu Optics. Please don't.

author: sync
author_rank: Frequent Contributor
timestamp: June 21, 2014, 02:39:52 pm
content: Quote from: pbgben on June 21, 2014, 02:23:58 pmSo you will sell a dozen worldwide.QuoteIt's basically a requirement for professional use. Non-ECC RAM increases the maintenance cost and for a lot of use cases it's not suitable because you can't trust the data. But there are other requirements like (on site) hardware support and software for managing the thing. So you are limited to the hobby users.

author: rob77
author_rank: Super Contributor
timestamp: June 21, 2014, 02:45:46 pm
content: my 2 cents...clustering R-pi boards is useless for real world applications... end of the story.why ?for parallel computing you need remote DMA (one node is able to write to the other node's memory) this is achieved by either InfiniBand, or if the latency is not SO critical then 10G Ethernet with remote DMA capability.if you think about hosting tens or hundreds of R-pi boards in a single case/rack - that makes sense - but that's NOT clustering

author: Monkeh
author_rank: Super Contributor
timestamp: June 21, 2014, 02:57:22 pm
content: Quote from: rob77 on June 21, 2014, 02:45:46 pmIt does?They're slow, power hungry for their performance, have no NICs..

author: rob77
author_rank: Super Contributor
timestamp: June 21, 2014, 03:02:09 pm
content: Quote from: Monkeh on June 21, 2014, 02:57:22 pmactually the R-pi hosting services are quite popular - therefore it makes sense (there is a demand for it). with the compute modules it would be harder in terms of connectivity , but still doable. if it has USB then IP over USB (works in linux - androids are tethered that way) would be a good candidate

author: Monkeh
author_rank: Super Contributor
timestamp: June 21, 2014, 03:03:35 pm
content: *shudder*

author: pbgben
author_rank: Contributor
timestamp: June 21, 2014, 03:05:22 pm
content: Quote from: tom66 on June 21, 2014, 02:33:36 pmThis is why I am asking to talk to someone that can, obviously I must do my research before hand. Also, just because I personally don't know how to do something doesn't mean I'll fail.Quote from: sync on June 21, 2014, 02:39:52 pmRaspberry Pi's are weak compared to a computer or ARM equivalent like the Odroid, but they are still successful.I can see the point you are making about the ECC, but you are seeing this wrong. You wouldn't use this for high accuracy math computes, and they don't even have to be used as a cluster. Each Pi is a separate device with its own CPU, Memory and Storage.Quote from: rob77 on June 21, 2014, 02:45:46 pmPeople have had an interest in clustering Pi'shttps://www.youtube.com/results?search_query=Rpi+clusterI see that its no good for advanced use, and thats the reason real clusters cost a s**t ton of cash to build and maintain. This setup is for people that need the features of a distributed compute cloud for a low cost.Since the first time this idea popped into my head, I have looked deeper into the practicality and uses, I have noticed that, as you said, they would be used more for their density/u then the compute-ability, I see this as more of a system for learning and low power distributed cloud set-ups.

author: theatrus
author_rank: Frequent Contributor
timestamp: June 21, 2014, 03:06:32 pm
content: Quote from: rob77 on June 21, 2014, 03:02:09 pmThe RPi already uses a USB to Ethernet chip. It's also dog slow - any other SoC with onboard Ethernet will have better network performance.

author: tom66
author_rank: Super Contributor
timestamp: June 21, 2014, 03:07:51 pm
content: The Compute Module is about $30 and a Raspi Model B board is $35, it is going to be cheaper to use separate Pi boards when you account for the necessary ethernet interface IC (if you so insist on a Pi cluster in the first place.)

author: pbgben
author_rank: Contributor
timestamp: June 21, 2014, 03:12:55 pm
content: Quote from: tom66 on June 21, 2014, 03:07:51 pmThe model B board is bulky, you don't need and video/audio/gpio/etc on the board for a compute setup. How much room would they take up compared to the compute modules

author: rob77
author_rank: Super Contributor
timestamp: June 21, 2014, 03:20:32 pm
content: Quote from: theatrus on June 21, 2014, 03:06:32 pmif considering IP over USB then you would skip the Ethernet completely - using a software router/switch running on a more powerfull SoC (e.g. the one you are suggesting with the onboard ethernet) terminating all the IP over USB lines for the R-pi modules (couple of USB controllers with couple of USB hubs would be needed).the R-pi hosting is never used for network inntensive applications - therefore a high performance network connectivity is not needed at all. so a 1:16 aggregation (networking Soc with ethernet providing downstream IP over USB to 16 R-pi boards) would be still more than enough bandwidth for the R-pi boards.

author: Monkeh
author_rank: Super Contributor
timestamp: June 21, 2014, 03:24:01 pm
content: Quote from: rob77 on June 21, 2014, 03:20:32 pmAlternatively, one could replace the whole damn mess with one of these and get better performance:http://www.supermicro.co.uk/products/system/1U/5018/SYS-5018A-MHN4.cfm

author: rob77
author_rank: Super Contributor
timestamp: June 21, 2014, 03:32:42 pm
content: Quote from: Monkeh on June 21, 2014, 03:24:01 pmthe idea behind the R-pi hosting is to have separate standalone small systems rented to /owned by individual users. some users simply prefer the "feeling" of their own little hardware over a virtual server/container.for example if you run a DNS server for a single domain along with a mailserver with a traffic of 5 mails per day... then the R-pi is still a huge overkill for thatp.s. if you don't like the idea - then it doesn't necessarily mean the idea is bad or useless

author: pbgben
author_rank: Contributor
timestamp: June 21, 2014, 03:33:55 pm
content: Quote from: Monkeh on June 21, 2014, 03:24:01 pmThats not distributed computing though, is it. Nice CPU though.

author: Monkeh
author_rank: Super Contributor
timestamp: June 21, 2014, 03:35:06 pm
content: Quote from: rob77 on June 21, 2014, 03:32:42 pmWell, there are always odd people around.QuoteQuite true! The performance, power consumption, and MTBF are what make it bad.Quote from: pbgben on June 21, 2014, 03:33:55 pmNo, it's a sensible idea, not a load of cheap chips nobody wanted to buy shoved in a box, because hype.

author: pbgben
author_rank: Contributor
timestamp: June 21, 2014, 04:05:17 pm
content: Quote from: Monkeh on June 21, 2014, 03:35:06 pm70Ghz in <600Watts and in 1U Thats not bad for performance/power (Don't forget cooling)If anything, they are better in regards to failure, as multiple modules can fail with no major effect to the "cluster"People do want, Pi's sell don't they?http://raspberrycolocation.com/

author: rob77
author_rank: Super Contributor
timestamp: June 21, 2014, 04:13:27 pm
content: Quote from: pbgben on June 21, 2014, 04:05:17 pmforget the 600W in 1Uand the Ghz is useless toobetter to count them in standalone servers per 1Uthe fact that you could physically fit a certain amount of those modules into a 1U case doesn't mean it's possible - not at all ! you have to consider all the supporting infrastructure around the modules. so basically divide the number of modules by 2 or 3 and you'll end up with a more realistic figure

author: Monkeh
author_rank: Super Contributor
timestamp: June 21, 2014, 04:13:56 pm
content: Clock rate is not a measure of performance.Anyone else want a stick to go at this horse with?

author: sync
author_rank: Frequent Contributor
timestamp: June 21, 2014, 04:18:59 pm
content: Quote from: pbgben on June 21, 2014, 04:05:17 pm

author: rob77
author_rank: Super Contributor
timestamp: June 21, 2014, 04:22:38 pm
content: Quote from: Monkeh on June 21, 2014, 04:13:56 pmof course it's not... a cpu's clock rate is the measure how good the design of the core is and how good the silicon manufacturing process isthe only valid measure is some kind of real-world end-to-end computing test (the CPU itself is just one piece of the puzzle).

author: linux-works
author_rank: Super Contributor
timestamp: June 21, 2014, 04:43:09 pm
content: I have a few rasp pi's and beaglebones, as well. I plan to buy a few more and put them into some kind of unified box, single psu, backplane to select ethernets or something (maybe just a crossbar/vlan switch).my use-case is network testing. I'm doing some SDN research and need a bunch of 'end nodes'. rpi's are good enough for that. they run a full linux distro (ipv4, v6, etc) and even openflow and openvswitch is there on them (I have yet to try but that's my plan). with ttcp and iperf, I've seen a reliable 80Mbps thru them and that's fair enough to do some flow testing.even a stack of mini itx boxes starts to take up room once you build a pool of nodes. the rpi/bbb form factor is great if you need to do some non-VM based network testing. compute power on these sucks big-time but they're fine as fanless and small end-nodes.

author: sync
author_rank: Frequent Contributor
timestamp: June 21, 2014, 04:51:25 pm
content: Linux-works, would you buy an 1U ras pi housing for a few 100s dollars alone for this?

author: pbgben
author_rank: Contributor
timestamp: June 21, 2014, 04:54:50 pm
content: Quote from: rob77 on June 21, 2014, 04:13:27 pmI see, my mistake.

author: linux-works
author_rank: Super Contributor
timestamp: June 21, 2014, 05:09:32 pm
content: Quote from: sync on June 21, 2014, 04:51:25 pmno, I don't want a rackmount, actually. it will sit on my desk or nearby. being low noise, I don't have to put this into a lab, I can have a 'mini lab' on my desk. that's part of the attraction.it may likely be more vertical than horizontal, but I'm not ready to think about physical form factor quite yet.for cost, I can make a case from plexiglass and have it custom cut to size (I have access to a laser) and so it will cost very little, for me.

author: josem
author_rank: Regular Contributor
timestamp: June 21, 2014, 05:22:40 pm
content: Forget clusters, that's so 90's Slashdot.Stick one Pi module inside each Solar Roadaways tile. That's your power sorted then just add a few connections and you'll have a Solar Raspberry Pi IoT CoR (Cluster on Roadway) system.Imagine the possibilities. Smart environment friendly roads AND a climate simulation platform!

author: sync
author_rank: Frequent Contributor
timestamp: June 21, 2014, 05:27:44 pm
content: Quote from: linux-works on June 21, 2014, 05:09:32 pmSo you are not a potential customer for pbgben's idea. I think most users of a small scale ras pi "cluster" will build their case them self instead of buying an expensive 1U frame.

author: rob77
author_rank: Super Contributor
timestamp: June 21, 2014, 05:28:06 pm
content: Quote from: josem on June 21, 2014, 05:22:40 pmi hope you're busy with preparing your kickstarter and indiegogo campaigns for this !i bet you'll outperform (in terms of collected funds) the solar roadway guys !

author: rob77
author_rank: Super Contributor
timestamp: June 21, 2014, 05:31:33 pm
content: Quote from: sync on June 21, 2014, 05:27:44 pmpotential customers would be the companies/individuals running the R-pi hosting services - but those would rather pay a group of engineers ( read engineering students - cheap workforce) to design and develop the platform for them

author: sync
author_rank: Frequent Contributor
timestamp: June 21, 2014, 05:38:14 pm
content: Quote from: rob77 on June 21, 2014, 05:31:33 pmHow big is the ras pi hosting market?

author: rob77
author_rank: Super Contributor
timestamp: June 21, 2014, 05:43:20 pm
content: Quote from: sync on June 21, 2014, 05:38:14 pmdoes it matter ?i already answered your next question in my previous postno it's definitely not a good candidate for a crowd funding project

author: linux-works
author_rank: Super Contributor
timestamp: June 21, 2014, 05:47:57 pm
content: no one would seriously 'host' on a rpi.the only thing these are good for is experimenting, playing, learning.if you plan to do serious and reliable thruput with these very underpowered boards, you are in for a surprise.a rack of atoms (AMD does this, or did this with their buying of the company sea-(something); I forget that name but they did high density of atoms meant for rackmount and datacenter use). it was power efficient and had mgmt tools all set for it.for me, being able to keep costs down and use 'commodity' cheap fanless linux 'blades' is a plus, but I would only use them for informal testing and experimenting.remember: the usb system (and ethernet) on the rpi is pathetic. I see constant syslog messages about dropped events. this is NOT a reliable platform, folks. its a hack. but its cheap and for some things, its ok enough.

author: SirNick
author_rank: Frequent Contributor
timestamp: June 24, 2014, 09:14:52 pm
content: I've designed a new multi-tenant hardware hosting platform called Blueberry Cobler. Looking for engineers to create the PCBs. And ICs. And chassis.Anyone in for a Kickstarter campaign?(I keed, I keed.)

author: fusionmkx
author_rank: Newbie
timestamp: November 07, 2014, 02:19:29 am
content: Been thinking about creating a cluster myself, some ideas were being toyed here,http://www.raspberrypi.org/forums/viewtopic.php?f=98&t=74447120 modules in a 1U rack. Well, it could be even denser and some water cooling channels to cool all the modules. Since it's sodimm form factor, you could probably use off-the-shelf heatspreaders.

author: blackice504
author_rank: Contributor
timestamp: November 13, 2014, 12:45:49 am
content: This is going to be really short and i will try to be sweet.Any SOC that is on the market does not even come close to the servers that are in Data centers especially the new onesExample of a Production web server "specs of single server" (these are normally in a cluster so depending on cluster size x the info by the following specs).2 x E5-2697V3 << latest socket LGA2011-314 Cores "28 Threads"9.6GT/s QPI2 x Quad Channel DDR4 "per cpu 1.5TB RAM"68GB/s40 PCIe 3.0 Lanes.145W10Gb Network "keeping in mind i have seen 40Gb and 100Gb" << just not common down under.SSD Raidyou got a shit load of bandwidth so much that it would make that cluster of Raspberry PI look like an old Atari or C64.Plus software i don't think Vmware / Citrix / Oracle / M$ / KVM. would work out of the box on the PI sure it could be modded ect but in hosting everything is about deployment and Bandwidth as well as low latency.Then keep in mind this machine is just 1 out of a rack that is clustered so that makes the PI even more useless for this application not to mention Processing Power per 1, after all there is a reason why nearly every data center uses on brand Intel. "sorry if your a fan of AMD or what ever but Intel Dominates the Server Market"Intel Costs more cheaper to run, more bang per watt and more bandwidth.Stuff to look forward to Intel and AMD and Nvidia are all set on an idea to make a super computer that will fit in the palm of your hand but even then i am sure that a desktop / server will have more power as its larger , has more bandwidth "more channels" ect.So in short sorry its not going to happen.

author: pbgben
author_rank: Contributor
timestamp: May 30, 2015, 08:09:45 am
content: Looks like someone else hase created a similar product...https://www.scaleway.com/features/

author: linux-works
author_rank: Super Contributor
timestamp: May 30, 2015, 05:04:07 pm
content: there was a company SeaMicro that tried this with intel atom chips, maybe 5 years ago. I had an interview there twice: both before amd bought them and after. funny, since they started off with atom chips for low power and then amd bought them (lol). they offered high density 500 cpu systems (iirc) that were mostly for i/o, thinking that business apps and webserver apps were more io bound than compute bound, at least many of them were. so, they tried to build configurable clusters of atom cpus with 'patchable' fabric, io, compute, backplanes, etc. kind of a neat idea, really.but they never made a big enough impact and they went out of business (amd killed the project, iirc).they had some serious pcb's, power supplies, chassis and over 100 people there at one time and they could not sustain this concept long-term, even with amd's funding!the concept of a pi cluster is for fun and learning, but NEVER more than that. sad but true; this 'imagine a beowulf cluster of ...' meme never wants to die, but other than for a proof of concept, its still not an efficient (cost or otherwise) way to build a cluster.

author: pbgben
author_rank: Contributor
timestamp: January 27, 2016, 06:36:00 am
content: A "Cluster" of the latest Pi'shttp://www.cnx-software.com/2016/01/22/this-is-what-a-16-raspberry-pi-zero-cluster-board-looks-like/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+cnx-software%2Fblog+%28CNXSoft+-+Embedded+Software+Development%29

author: Monkeh
author_rank: Super Contributor
timestamp: January 27, 2016, 04:41:17 pm
content: What a ridiculous waste of money.

author: suicidaleggroll
author_rank: Super Contributor
timestamp: January 27, 2016, 07:00:07 pm
content: If you read the details, they did NOT develop that board for clustering/HPC, they built it to rapidly program and test their deployable sensors. It lets them program and test 16 boards at a time in a convenient and self-contained package, instead of having to individually wire them up one by one.QuoteDepending on how many of those sensors they plan to build/sell, that "clustering" platform could easily end up paying for itself in reduced labor costs. Of course the "fan boys" are jumping on it and begging to be able to buy it as a compute cluster, but that's not the company's fault.