High MIPS/watt clusters - Page 1
Forum section: electronics / general

author: Fulcrum
author_rank: Regular Contributor
timestamp: April 22, 2016, 07:14:31 pm
content: Daves new video on using a Raspberry Pi to hunt for aliens was interesting. I was especially surprised at how efficient the Raspberry Pi processor is when it comes to how many operations per watt it can perform. If you haven't watched the episode, it had a MIPS/watt 5-8 times that of Daves Xeon-based computer! This got me thinking of something. Would there perhaps be a market for efficient, low-power computing units meant to perform this kind of distributed computing?I know you can buy a bunch of Raspberry Pis and hook them up to make a cluster, but would it not be easier to just buy a PCB with a bunch of processors on it? It would also be cheaper, as you don't need to pay for all the other stuff that comes with the Raspberry Pi. Naturally it could be modular so you could buy many modules and stack them on top of each other, for example having each PCB/module come with a 3x3 grid of processors. I realize the market for something like this might not be too big. The Raspberry Pi got popular not for being super efficient but for being small, cheap, and versatile. A computing module like what I have described would only be good for efficient scalable computing.

author: stmdude
author_rank: Frequent Contributor
timestamp: April 22, 2016, 07:23:59 pm
content: The datacenter people are way ahead of you.Google around for "ARM Servers", and you'll see that everyone and their grandmother are designing chips and products for it.The reason it hasn't happened quite yet is that most ARM chips lacks high-speed interconnects (PCI-X, etc). But Renesas, AMD and a few others are rectifying that.

author: Fulcrum
author_rank: Regular Contributor
timestamp: April 22, 2016, 07:52:05 pm
content: Quote from: stmdude on April 22, 2016, 07:23:59 pmOh, of course someone had thought of it! Damn other people, taking all the good ideas. Very interesting though, I can imagine the server people would be interested in high efficiency. They trade power usage for space usage though, don't they? Less power consumption per MIPS, but having to replace every Xeon with several ARMs. I suppose floor space is less expensive than electricity. And with all the pressure on companies to go green, it might weigh up for it some.

author: stmdude
author_rank: Frequent Contributor
timestamp: April 22, 2016, 08:14:22 pm
content: Quote from: Fulcrum on April 22, 2016, 07:52:05 pmWell, it's a combination of many things. Electric power being one of them.Floor-space isn't terribly important any more, as the infrastructure has involved to the point where you can build datacenters where land is cheap.Cooling is a big one that people usually don't think of.Imagine one single rack of servers. A typical server today is 1U (one inch) tall, has two Xeon E5s in it, and you can stack about 72 of those in one rack.Each E5 (smallest model) has a maximum TDP of 130W. 130*72=9360W. That's about 10KW of heat that needs to go away. For one rack.Another thing is that one server can only (without resorting to crazy tricks) serve just over 64000 simultaneous connections. For certain applications (chat servers, games, etc come to mind), this is very limiting.Price is also an issue. A Xeon CPU costs about $100 from Intel. You can get ARM chips for a lot less, even the ones meant for servers.

author: Fulcrum
author_rank: Regular Contributor
timestamp: April 23, 2016, 12:48:03 am
content: Quote from: stmdude on April 22, 2016, 08:14:22 pmWhat limits the number of connections? In my naivity I would expect it to scale linearly with the number of processors or racks you slap in your server.

author: kc8apf
author_rank: Regular Contributor
timestamp: April 23, 2016, 04:45:30 am
content: Quote from: Fulcrum on April 23, 2016, 12:48:03 amOne limitation is that TCP and UDP use a 16-bit port number. If your server has only a single IP, there are only 65535 unique destination ports per protocol. Of course, you can always add additional IPs to the NIC, add more NICs, do user-space packet switching, etc.

author: matseng
author_rank: Frequent Contributor
timestamp: April 23, 2016, 04:59:47 am
content: No, not really . TCP connections are specced "tuples" i.e combinations of source IP&Port and the destination IP&Port. You can have millions of connections into a server, is basically only the memory used by the IP stack that is limiting the number of simultaneous connections into a server - as long as they come from multiple clients.

author: kc8apf
author_rank: Regular Contributor
timestamp: April 23, 2016, 05:35:52 am
content: True. I was thinking about the cases I run into at work where you have load balancers and NAT'd clients.

author: NiHaoMike
author_rank: Super Contributor
timestamp: April 23, 2016, 06:59:17 am
content: Quote from: blueskull on April 22, 2016, 10:38:28 pmGPUs do even better if your workload can run well on a GPU. A GTX 970 is also around $300, but only uses 145W and delivers 3.5 TFLOPS single precision, for an excellent 24.1 GFLOPS/W. Granted, that's for the GPU by itself and the host machine it's installed in will contribute some overhead. (To be fair, the Xeon Phi suffers from the same problem.) For my machine, it ends up at 180W Folding with GPU only. (I actually no longer Fold with the CPU since that nearly doubles power usage for about 10% better throughput.) The overhead can be reduced by putting multiple GPUs in a PC, if you want to scale things up.The Pi 2 only uses 2.3W according to the video, which translates to 6.96 GFLOPS/W. Then you have to consider the overhead of the extra switch ports needed to actually connect the Pis to the network. For Pis used only for compute with very low inter node bandwidth requirements, maybe it's possible to use the A version (if one with an up to date CPU were actually released) and then make some more efficient way to network them. A low power FPGA serving as a backplane controller might work. Or maybe put the slave Pis into USB OTG and connect them to a master Pi. Or for very low bandwidth requirements, loop the UARTs or slave mode SPI around like Token Ring, possibly with a little multiplexing logic and watchdog timers to prevent one locked up Pi taking down the whole system.

author: bitslice
author_rank: Frequent Contributor
timestamp: May 05, 2016, 01:03:36 pm
content: Quote from: stmdude on April 22, 2016, 07:23:59 pmThe transputer addressed that issue 36 years agohttps://en.wikipedia.org/wiki/TransputerPCI is overly complicated as a chip to chip link.

author: EEVblog
author_rank: Unknown
timestamp: May 06, 2016, 03:32:18 am
content: Quote from: blueskull on April 22, 2016, 10:38:28 pmI'm interested in this.Would something like this Xeon Phi PCI card "just work" with BOINC?http://www.ebay.com.au/itm/Intel-Xeon-Phi-31S1P-Coprocessor-8-GB-57-Cores-1TFLOP-270W-Sealed-BC31S1P-New-/222098831440?hash=item33b61f3850:g:fVwAAOSwpDdVTyRri.e. plug it in and you get 57 cores on your Windows box?

author: helius
author_rank: Super Contributor
timestamp: May 06, 2016, 04:00:09 am
content: Quote from: EEVblog on May 06, 2016, 03:32:18 amNo, it isn't like that. You need to write applications specifically for the Xeon Phi (known by the compiler as the MIC). You can do that using parallel libraries like MPI or OpenMP, or write native applications for it. It is a x64, but with completely different extensions from the host processor.The board also requires firmware support, which is only present in specific server systems.

author: coppice
author_rank: Super Contributor
timestamp: May 06, 2016, 06:44:16 am
content: Quote from: blueskull on May 06, 2016, 04:15:46 amWhen I was looking at the cheap Xeon Phi cards last year I was surprised how few motherboards they will work with. They won't work with a lot of server and workstation motherboards, because of BIOS bugs, or may only work if you update to the latest revision of the BIOS first. I wouldn't get a Xeon Phi card without first confirming that it will work with the motherboard you intend to use.

author: suicidaleggroll
author_rank: Super Contributor
timestamp: May 06, 2016, 07:17:37 pm
content: Quote from: coppice on May 06, 2016, 06:44:16 amYep - I purchased one hoping it would work in my Supermicro X9SRH-7TF, and despite it having the option to exceed the 4 GB memory limit, it still wouldn't recognize it. I ended up popping it into a Supermicro SYS-1028GR-TRT where it works happily. The above posters are correct about binary compatibility. While it's technically x86, you can't just copy over an executable from your host system and expect it to run. You need to compile it specifically for the MIC, and even then the code needs to be massively parallel in order to scale well. Not to mention that the "P" in 31S1P stands for passive, you need a lot of airflow through your case to keep it cool, because it doesn't have any fans of its own.In my testing I found the 31S1P as a whole to be roughly equivalent to 4 of the 14 cores on an E5-2697 v3. I expected better, but that probably has to do with the type of processing I was doing in my test (mostly data analysis).