The search for a (CHEAP) supercomputer... - Page 1
Forum section: electronics / projects

author: cprobertson1
author_rank: Frequent Contributor
timestamp: April 28, 2017, 01:29:55 pm
content: Good Afternoon folks!DISCLAIMER - I am going to use the "CPU Clk spec" rather than the "CPU FLOPS spec" as it's an easier number to work with since I'm not just doing floating point operations: the test is a complex algorithm which runs assembly code in a virtual machine (inspired byCorewars,whoremembersthatone!?) - once the virtual machine is set up it only needs to run once: and you can run as many tests as you want in it, fairly quickly -a lacorewars! - the problem is that I don't think that directly translates into the CPU FLOPS spec =/Sorry in advance!I have an interesting project on the go just now - a code-generating genetic algorithm.It's fairly simple in practice - it generates some code, mutates it, and tests it - the best part about this process is subdivisible - once a process receives it's orders it carries out the tests and reports back the results: which is useful because it only runs on one core at a time (some dufus borked the hyperthreading code (Surprise!!! It was me)).At the moment it's running on a 4-Beowulf Cluster of Raspberry Pi (equivalent speed... of... ~3.2GHz?) - which works out at ~84 tests per second (~300'000 per hour)The upshot of this single-core behaviours is that it maxes out whichever core it's using, crunching the numbers as fast as it can - you can just split the job into separate parts and assign them their own processes (in separate cores) and voila! Works (more or less) equally well whether it's a single multicore CPU or multiple CPU on multiple boards networked together (obviously the latter has more losses due to larger bottlenecks in transfer speeds between the master and slave nodes)In short, there is a direct correlation between CPU clock speed and the number of calculations performed (with some losses due to inefficiencies summing together - just means it's not a linear correlation).But... I wantMORE POWER!I can obviously expand the size of the Raspberry Pi Cluster, maybe copying Dave's idea of using Orange Pi instead of the regular ones... but the problem is that the cost of these units isn't insignificant - and I'm convinced I can do better!At the moment I have a single (kinda crappy) 16-port ethernet switch - which is linked to the 4x Raspberry Pi: so in principle, I can run up to 16 RPi, which works out at about... ~11Ghz in total (plus some losses... okay, from now on just assume whenever I list a CPU speed that it includes "the losses from various stuff happening")So, that works out at £36/GHz - and I reckon I can do better than that!On the other hand, I've just got the results back from my local University whom I rented some time on their supercomputer from - that's not a very good sentence - that worked out at £~50 / 1000 core hours (which... I don't know how to convert that to GHz... but it's got a peak performance of 38 Teraflops peak performance - which might actually be cheaper than having my own miniature, super-low-end supercomputer (but with the disadvantage that I can't put it in my livingroom and use it to display pretty images of evolving code 24/7- BEST PICTURE FRAME EVER)Going down Daves route with the Orange-Pi gives me about £20/GHz - BUT I reckon I can do even better!With the Orange Pi, each unit comes withextra stuff- GPUs, USB busses, GPIO, and some of them even have integral WiFi! Score! - but, you're definitely going to be paying for that extra hardware - I,in principleneed the bare minimum to run tinylinux (I might not even need to go that far), meaning a CPU, RAM and Ethernet supportDoes anybody have any suggestions on how I can up the total processing power?I'd like to aim for £15-20/GHz if I could get that far - but I'm struggling to come up with any ideas that don't revolve around ASICs or FPGAs (the former of which are expensive and the latter of which involves programming my own CPU which doesn't sound fun... especially considering I don't know diddly-squat about FPGAs)Many thanks!

author: CJay
author_rank: Super Contributor
timestamp: April 28, 2017, 01:37:59 pm
content: Curious as to how that stacks up against the offering from Amazon (even the free offering)?

author: CM800
author_rank: Frequent Contributor
timestamp: April 28, 2017, 01:40:58 pm
content: Hello,I'd suggest checking out the NanoPi NEOAllwinner H3 (quad 1.2GHz)for under $8 (256MB) or $10 (512MB)http://www.friendlyarm.com/index.php?route=product/product&product_id=132

author: daqq
author_rank: Super Contributor
timestamp: April 28, 2017, 01:55:38 pm
content: You could investigate the options of renting computing power from companies - you can 'rent' any number of Amazon or Google (or other cloud) solutions, some of whom can pack a pretty punch.For a simulation I rented some monster computer from them, left it running for an hour, crunching numbers.

author: Tom45
author_rank: Frequent Contributor
timestamp: April 28, 2017, 02:06:09 pm
content: If your code can be ported to OpenCL you could then run it on highly parallel video cards.Also see:https://petewarden.com/2014/08/07/how-to-optimize-raspberry-pi-code-using-its-gpu/

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: April 28, 2017, 02:17:32 pm
content: Quote from: CM800 on April 28, 2017, 01:40:58 pmAnd the Orange Pi Zerohttps://aliexpress.com/store/1553371

author: NiHaoMike
author_rank: Super Contributor
timestamp: April 28, 2017, 03:28:44 pm
content: Put together a cheap Xeon Phi setup? Or port the code to run on GPU?If you just want a lot of small, cheap nodes, be on the lookout for super cheap smartphones.

author: macboy
author_rank: Super Contributor
timestamp: April 28, 2017, 05:28:54 pm
content: Maybe not what you are looking for but, this winter I put together a Xeon system with 16 cores (32 threads) at 3.2 GHz, and it cost me roughly $1k CAD (~£600). The CPUs are 'officially' 2.9 GHz, turbo up to 3.8 GHz, but will run flat out at 3.2 GHz with all cores fully loaded as long as they don't overheat (they don't). So 16 x 3.2 is 51.2 GHz by your metric. 600 £/51.2 = 600/51.2 = 11.7 £/GHz. If you account for a 30% increase in available CPU power with hyperthreading, then that's£9/GHz. And you'll find that "1 GHz" worth of Xeon CPU will dofar morework than "1 GHz" of RPi CPU. This is true even if you just use basic x86 instructions, but even more true if you can optimize for SSE, etc.The machine consists of a pair of E5-2690 CPUs (8C/16T each) on an ATX form factor Supermicro X9DRL-iF dual CPU motherboard, eight sticks of 8 GB Samsung DDR3R (registered, ECC) RAM (quad channel each CPU), two giant SNK-P0048AP4 heatsinks, plus the usual bits like a beefy power supply, case, fans, SSD, etc. For very basic server stuff you can get away with the on-board video, but to even use it as a basic workstation you'll want a 'real' video card, even if a low end one. Everything was purchased used for much less than the original retail price. These used server components have a weak resale market so the prices are really low. E.g. $120 USD for the eight sticks of 8 GB RAM. The motherboard is the biggest ticket item. If you can deal with a strange large form factor then you have a better chance at a bargain; I paid more for standard ATX size.With this, you can also encode 4k video faster than realtime and other similar feats that a RAM-starved RPi farm can't do.

author: Marco
author_rank: Super Contributor
timestamp: April 28, 2017, 07:43:12 pm
content: Unless the genetically optimized program is supposed to operate on wide vectors I don't quite see how a GPU or Xeon Phi is supposed to help.

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: April 28, 2017, 08:07:23 pm
content: Quote from: evb149 on April 28, 2017, 05:58:14 pmThat's very true. 100% cpu load on all cores at once plus no heatsinks and cpu throotling will kick in very very soon on the RPis and OPis.

author: NiHaoMike
author_rank: Super Contributor
timestamp: April 29, 2017, 01:58:25 am
content: Here's a pretty large list of cross platform benchmarks:http://www.7-cpu.com/Per core, modern x86 CPUs are not that much faster than modern ARM cores. A single A53 (used in the Raspberry Pi 3) scores 880 and 1600 and a single Kaby Lake core 4900 and 4700. It would be easy to build a desktop with 10x the CPU power of a Pi 3 but 50x not so much.If you can find cheap but decent smartphones (requires patience), those are likely to do very well in terms of performance per watt and performance per dollar for tasks that do not run well on GPUs. For example, I have a cluster of 4 quad core smartphones (three Moto Es and a ZTE Speed) that cost $70 and use less than 10W total mining altcoins. A x86 system to emulate those 4 smartphones will definitely cost more than $70 (at least when new) and use a lot more than 10W, especially if there's ARM on x86 emulation to deal with.

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: April 29, 2017, 08:09:44 am
content: Quote from: evb149 on April 29, 2017, 03:07:26 amYes the billions of transistors in an Intel CPU are there for a reason, but don't overlook that 800 freedom kopecs buy you more than 100 Orange Pi Zeros. That's 400 pretty decent ARM cores @ 1.6GHz.

author: brucehoult
author_rank: Super Contributor
timestamp: April 29, 2017, 09:43:03 am
content: Quote from: Marco on April 28, 2017, 07:43:12 pmGPUs are a very different kind of parallel operation to SIMD like SSE or AVX. With those everything has to run exactly the same instruction in the calculation, every time.In a GPU you write the program as if each calculation is on a different CPU, the same as for ARM or x86. You can have if/then/else, loops, whatever. The hardware runs the threads in groups (typically 32) called a "warp". All the threads in one warp run the same instruction at the same time. If some of the threads don't want that instruction then they sit idle for a while.For example, if your code says 'if (x > y){mx = x; foo--;} else {mx = y; foo+=2;} printf("%d", mx);' and x is in fact greater than y in 90% of the threads. They will all run the test together. And then 90% of the threads will do "mx = x; foo--" while the other 10% sit idle. And then 10% of the threads will do "mx = y; foo+=2" while the other 90% sit idle. And then all 100% of the threads will together do the printf. (yes, OpenCL has printf)If you have a loop and some of the threads finish the loop quickly but others are still going ... the ones that finished sit idle until the others exit from the loop.So you should try to arrange your code so that as high a percentage of each group of 32 threads as possible are taking the same execution path. But if they diverge from time to time it's ok, as long as they join together again soon.The top modern graphics cards such as the NVIDIA 1080 TI now have about 3584 CPU cores in 112 warps running at about 1.5 GHz, for $600. Even if you have really bad luck with idle threads in a warp, that's still a *lot* of pretty fast CPUs for $600. And if you get lucky or write your code carefully .. it's 32x more.A bytecode interpreter should run tolerably well on a GPU. It's not ideally suited, by any means, but it's not *horrible* You're probably going to drop back to one thread in a warp at a time running the actual guts of the bytecode, but the fetch and decoding of the bytecodes should happen with all 32 cores in parallel. If your interpreter loop has say 90% of the instructions for housekeeping and 10% running the actual operation (pretty typical) then I'd guess you'd end up using on average about 500 - 800 of the 3584 cores at any given clock cycle. That's a lot of MIPS.For sure it will be much better than $600 worth of Raspberry Pis :-)

author: cprobertson1
author_rank: Frequent Contributor
timestamp: April 29, 2017, 02:03:19 pm
content: Oooft! Lots of great info here!At the moment the program can be set to compile CoreWars code, OR my own system inspired by it - though they work in the same way - my way is a little more modern BUT requires alittlemore processing power (i.e it's slightly less efficient). As a result, the flow chart below uses "pMARS" as the assembly compiler/tester instead of my own system (and there are more efficient MARS compilers out there anyway - I haven't optimised that side of things yet!)Relating to the program flow:The code all runs under anoverseerapplication (python) which starts off all the other modules and then acts as a watchdog timer and memory manager - deleting old files as required.Theoverseerstarts thecompiler, using a series of initial parameters - this generates a series of test-code, each in it's own text file (typically <500Bytes, limited to about 4.2kB) - how these are stored depends on the config file (they can be stored in RAM or on a particular drive)Theoverseerthen starts theschedulerwhich splits up the population for testing, again based on the configuration - deciding which cores to start processes in.On top of that, (but still under theoverseer) is thecontroller, which controls the data flow between the master node (which does everything except testing) and any slave nodes. I know that's not the right terminology but you understand exactly what I mean so I'm not going to stop using itThecontrollerjust sends test-orders and any required code-to-be-tested to the slaves, and receives the results back (which it in turn reports back to thecompilerwhich then "breeds" the test-code) and the process restarts.The test just involves generating a virtual memory core with assembly commands in it and then executing the code while alternating between a number of pointers: after a number of conditions are met (usually the number of cycles) a scoring algorithm is run and the "winner" reported back to thecontroller: there isn't particularly hard on the PC - in fact, CoreWars was originally released in 1984 and it ran fine then- the RAM requirements are minimal - each simulated "Core" requires 448KB for the data in the core, and another 50kB for the supporting code - it's tiny - which is what I rely on when I run as many tests as I can!BUT because of this it's quite a lot of high-level stuff - I'm not sure it'd be easy to optimise to run on a GPU =/ There are some optimisation tweaks I've done so far and I'll optimise it a little more as I go through it.ThoseQuote from: GeorgeOfTheJungle on April 28, 2017, 02:17:32 pmAh, that NanoPi might be exactly what I'm after: the cost:power ratio is pretty high - the Orange Pi Zero has higher power consumption and costs more - so the Neo may be better for my application in terms of cost.As for cooling (and noise pollution) - I'm thinking of simply submerging them in a tank of mineral oil (and maybe use an aquarium pump to add some flow to the oil) - provided it's low oxygen oil it shouldn't cause any corrosion problems.16x NanoPi Neo works out at ~16 GHz of processing (including losses - it'll likely be higher) and that should be fairly expandable - so I can start slow and ramp up the power over time.Not sure what I'll go for yet though - still researching it!In the meantime I have a MiniPC from 2003 that I'll use for running 24/7 in the living room while I work on upping the processing power - it's a little slow but it will at least look pretty!

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: April 29, 2017, 04:11:32 pm
content: To prove that at least for some things, such as this useless microbenchmark, $800 of arm cores are better/faster and less expensive than an intel CPU, here's a short program that spawns 4 threads. Each thread creates an own 16MB buffer and receives a common/global 16MB buffer, then averages every byte and saves it in its own buffer and then returns it.My thousands of freedom kopecks i7 MacBook Pro gives:jorge@unibody:~/kk$ gcc -O0 threads.c -o threadsjorge@unibody:~/kk$ time ./threadsreal 0m0.113suser 0m0.313ssys 0m0.052sAnd my $8 Orange Pi Zero:pi@orangepi:~$ gcc -lpthread -O0 threads.c -o threadspi@orangepi:~$ time ./threadsreal 0m0.491suser 0m1.700ssys 0m0.170sThat's about 1/5 the speed for much less than 1/100 the price.Code:#include <pthread.h>#include <stdlib.h>#define BUF_SIZE 16*1024*1024pthread_t t1, t2, t3, t4;void* thread_proc (void* arg) {int i= BUF_SIZE;char* buffer= malloc(BUF_SIZE);while (i--) buffer= (((char*) arg) + buffer) / 2;return buffer;}void create_threads (char* buffer) {pthread_create(&t1, NULL, thread_proc, buffer);pthread_create(&t2, NULL, thread_proc, buffer);pthread_create(&t3, NULL, thread_proc, buffer);pthread_create(&t4, NULL, thread_proc, buffer);}void join_threads () {void** r;pthread_join(t1, r), free(*r);pthread_join(t2, r), free(*r);pthread_join(t3, r), free(*r);pthread_join(t4, r), free(*r);}int main (void) {char* buffer= malloc(BUF_SIZE);create_threads(buffer), join_threads();free(buffer);return 0;}

author: daybyter
author_rank: Frequent Contributor
timestamp: April 29, 2017, 10:27:31 pm
content: https://www.eevblog.com/2016/10/15/eevblog-934-raspberry-pi-supercomputer-cluster-part-1/

author: JuanAG
author_rank: Newbie
timestamp: April 30, 2017, 03:56:34 am
content: I am trying something similar and the pis are to big for me, really i doesnt need much stuff from it so i am trying with CPUs instead, this is very similar to the broadcom the pi hashttps://www.olimex.com/Products/Components/IC/A64/It is only 5€, more or less 5$If the software can run parallel is the way to go, you put 20 or 30 on a PCB and you will get a huge performance for a little cost and with few watts of energy, believe me when i say that an x86 pc or server cant match this, for the same price with the arm you always will get more performance and less waste of energyAnd up there someone has said that 1 GHz of an arm CPU is less than a 1 GHz of an x86, no, it is not true, most of the operations take the same amount of cycles and the arm a53 has SIMD instructions as well, they dont call sse or avx but exists and are for all you can imagine, add, sub, div, mul and with 128 bits, 64 bits (the native bits), 32 bits and more, from the type you choose you will have the numbers of pararell operations, if you choose 128 you will get half of 64 and so on

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: April 30, 2017, 07:04:30 am
content: Quote from: JuanAG on April 30, 2017, 03:56:34 amI have to disagree there ^^^, the billions of transistors in an Intel CPU are there for a reason: performance. ARM cores are much simpler to save energy (and that's why they are in every smartphone in the planet). If you ever wanted an ARM core to perform as well as an Intel you'd have to put in it nearly as many transistors as an Intel has but then would as well require as many watts.

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: April 30, 2017, 07:27:23 am
content: Quote from: cprobertson1 on April 29, 2017, 02:03:19 pmIIRC those nanopis don't have proper voltage regulator PMICs nor proper pcb designs so they overheat too easily, ISTR I've read that in the armbian forums (ARMBian is the best distro for these RPi "clones").These are the only urls I have at hand right now, but there are others IIRC.https://forum.armbian.com/index.php?/topic/1351-h3-board-buyers-guide/https://linux-sunxi.org/FriendlyARM_NanoPi_NEO#Voltage_regulators_.2F_heat

author: brucehoult
author_rank: Super Contributor
timestamp: April 30, 2017, 10:59:49 am
content: Quote from: GeorgeOfTheJungle on April 30, 2017, 07:04:30 amThat would be true if ARM and x86 were similar instruction sets, but they are very different and x86 is *much* harder to decode and requires more transistors, more area, more power.In the actual execution engines, right, there should be little difference at equivalent performance levels.So it all depends on the ratio of the size of the decode part to the size of the execution part.The entire point of using GPUs for "compute" tasks is that the instruction decode (and execution control) is shared between 32 or so CPU cores. Current GPUs have anywhere from 256 to nearly 4000 cores, running at up to 1.5 GHz or so. That shows how expensive decode and control is compared to ALUs and registers.On the highly out-of-order implementations in high end x86 since the Pentium Pro it turns out that decode is a relatively small percentage of the total circuitry. However, while these OOO implementations give you more performance, they don't give you more performance in proportion to the transistors you spend on them. If you need single-threaded performance above all then it's the only way to go.But if you want overall throughput on a parallelizable task, or something that is inherently many individual tasks, such as web serving, then it works out better to use the same number of transistors to make a lot of simple pipelined single issue (maybe dual at most) cores. Like in the Xeon Phi. Like in the Cavium ThunderX with 48 Aarch64 cores running at 2 GHz on a chip -- you can rent ARM64 cloud servers with 96 cores and 12 GB RAM for $0.50/hour herehttps://www.packet.net/bare-metal/servers/type-2a/On those kinds of masses of simple cores, the expense of x86 decode does seriously hurt you.Even the accumulated ARM/Thumb/Thumb2/Jazelle/ThumbEE mess hurts you quite a lot. Those Cavium CPUs are 64 bit *only*. No legacy 32 bit ARM instruction sets.RISC-V is looking good for the near future too. It's pretty equivalent to Aarch64, but with Thumb2 code density, and specifically designed to be cheap and fast to decode. 1.6 GHz quad core 64 bit SoCs more or less like the A53 are expected out around the end of this year, and people are prototyping 1000 core SoCs in the lab.

author: Fungus
author_rank: Super Contributor
timestamp: April 30, 2017, 12:06:56 pm
content: Quote from: GeorgeOfTheJungle on April 29, 2017, 04:11:32 pmThat's a meaningless test if the OP is interested in double precision floating point.And... there's the problem. This question can't be answered without knowingexactlywhat the program does.eg. For single precision floating point a GPU will destroy anything else in its path. Those things can easily have thousands of cores.For double precision floating point a desktop CPU will easily win against ARM chips......up to a point. If you're allowed to usehundredsof ARM chips then eventually you'll overtake the desktop.

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: April 30, 2017, 02:55:14 pm
content: Quote from: Fungus on April 30, 2017, 12:06:56 pmRight you are. There you go:My thousands of freedom kopecks i7 MacBook Pro:jorge@unibody:~/kk$ gcc -O0 threads.c -o threadsjorge@unibody:~/kk$ time ./threadsreal 0m0.719suser 0m2.238ssys 0m0.098sThe $8 Orange Pi Zero:pi@orangepi:~$ gcc -lpthread -O0 threads.c -o threadspi@orangepi:~$ time ./threadsreal 0m9.754suser 0m7.940ssys 0m25.370sOr about 14x the speed at more than 100x the price.Code:#include <pthread.h>#include <stdlib.h>#include <stdio.h>#define BUF_SIZE 16*1024*1024pthread_t t1, t2, t3, t4;double* nu_buffer () {double* buffer= malloc(BUF_SIZE);int i= BUF_SIZE/8;while (i--) buffer= (random()-random())/(random() + 1.0);return buffer;}void* thread_proc (void* arg) {int i= BUF_SIZE/8;double* buffer= nu_buffer();while (i--) buffer= (((double*) arg) + buffer) / 2.0;return buffer;}void do_threads_stuff (double* buffer) {pthread_create(&t1, NULL, thread_proc, buffer);pthread_create(&t2, NULL, thread_proc, buffer);pthread_create(&t3, NULL, thread_proc, buffer);pthread_create(&t4, NULL, thread_proc, buffer);void** r;pthread_join(t1, r), free(*r);pthread_join(t2, r), free(*r);pthread_join(t3, r), free(*r);pthread_join(t4, r), free(*r);}int main (void) {double* buffer= nu_buffer();do_threads_stuff(buffer);//free(buffer); return 0;}

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: April 30, 2017, 03:06:44 pm
content: Quote from: brucehoult on April 30, 2017, 10:59:49 amTrue, but they also have much larger caches, built-in crypto/AES engines, turbo-boost, wider and faster buses, hyperthreading, virtualization extensions, etc. All that counts as well, an ARM with all these bells and whistles would very likely need *nearly* as many transistors as the Intel.In any case those aren't my words, I heard Sophie Wilson herself say exactly that, and I'm just repeating her words as a parrot hahaha.

author: NiHaoMike
author_rank: Super Contributor
timestamp: April 30, 2017, 04:34:21 pm
content: Quote from: brucehoult on April 30, 2017, 10:59:49 amI remember reading that Apple plans to phase out 32 bit support from iOS. About what % advantage would that yield?

author: dom0
author_rank: Super Contributor
timestamp: April 30, 2017, 05:11:36 pm
content: Pretty much the cheapest option is to find either complete datacenter pulls or reassamble from pulled parts of the sandy/ivy bridge generation that are being phased out since a year and a half or so. The CPUs (E5-2650 is the by far most common SKU) dipped as low as 20 $ each (8C/16T 2.8 GHz or so), pretty much everything else on such a system will be more expensive (even the case).Quote from: brucehoult on April 30, 2017, 10:59:49 amYou have some good info there and some completely wrong.64-bit ARM cores tend to have _four_ frontends for the different ARM ISAs. Only some specialty cores can afford to implement only one decoder, e.g. Apple and Cavium do that. Modern x86 designs (Intel, AMD Zen) aremassivelyout-of-order cores (~300 instructions in-flight), usually with an 8-way backend ("ports", "8-issue") and a massive register file, neither of which is done by any ARM core (if you would, you'd need pretty much the same power as everyone else, big surprise). The ISA implemented by a CPU plays a rather minor role in it's efficiency and power consumption today.Similarly if you look at a POWER or SPARC core the decoder takes a large part of the logic transistor budget; in almost all CPU cores -regardless of ARM/x86/POWER/SPARC- nowadays the decoder (the parts you can identify as one) is usually the largest logic part (even on the floor plan).The reason for the increased ALU efficiency in GPUs comes from putting many ALUs under the control of one frontend and one scheduler, and having exactly no dynamic optimization, be it register renaming, OooE or branch prediction (any branching, actually; GPUs do not branch, they mask) in the scheduler. By running the entire warp in lockstep a lot of circuitry is saved. This works very well for some workloads and works absolutely not at all for others. Thus a comparison to CPUs simply makes no sense.Similarly simple cores are totally fine for linear, ALU heavy workloads and are applied there (PEZY, A2, Fujitsu), but suck balls at general purpose application.QuoteThis maybe works on paper, but has never worked out in practice, be it Niagara / Ultrasparc Tx, or Cavium; the former just had too poor performance, though it actually did work in some niche applications; the latter is plagued by performance issues and bugs all over the chip. Cavium said the next generation will surely fix them all. Notice how a few years ago many vendors made a big noise abou AArch64 in the data center - they're all pretty silent now...Oracle tells that the newest shiniest massively threaded SPARC cores are much much better now, but I bet that in general purpose use (i.e. not running a database using all those shiny DB-ISA extensions) any x86 core still bashes it's head in with ease, as always.

author: dom0
author_rank: Super Contributor
timestamp: April 30, 2017, 05:18:26 pm
content: "That CISC decoder is so complex and power hungry and needs so many transistors" is pretty much just 90s RISC propaganda these days. Heck, itwasthe 90s when Intel, AMD and some other companies(!) showed off that the ISA on the frontend doesn't matter and that the popular marketing, even academic, opinion that x86 will never be as high performance as RISC (POWER/SPARC/Alpha), because it's "literally impossible to use pipelining" or make procedure calls fast or any other disproven "fact", is bull*****.The truth is that it doesn't matter and that's precisely the reason why all RISC architectures were flushed down the drain by 2000 in the mainstream server and workstation markets. A bit later all long-pipeline µarchs were flushed after them (some pun intended). Pipe didn't clog from those, either.

author: dom0
author_rank: Super Contributor
timestamp: April 30, 2017, 05:22:56 pm
content: Quote from: GeorgeOfTheJungle on April 30, 2017, 02:55:14 pm-O0, seriously? Are you trying to make a joke out of yourself?Pretty sure random() will take a lock on each call, the program probably spends most of the time contending over that lock, not doing the calculations.Edit: And, indeed, using random_r correctly, not contending one lock from all threads, speeds this up by a factor 15.

author: dom0
author_rank: Super Contributor
timestamp: April 30, 2017, 05:28:15 pm
content: By the way, all cheap ARM SBCs are based on toy SoCs with toy cores like the A53, which is a low performance low power (low efficiency) core, with a low performance memory system and laughable I/O capability. They're today's bottom-of-the-barrel chips in the ARM world.

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: April 30, 2017, 06:16:04 pm
content: Quote from: dom0 on April 30, 2017, 05:22:56 pmFeel free to fix it! My broken code is free and open source! Hahaha.Edit: I see no random() in the main thread loop, so ?Code:while (i--) buffer= (((double*) arg) + buffer) / 2.0;

author: dom0
author_rank: Super Contributor
timestamp: April 30, 2017, 08:17:04 pm
content: All threads execute nu_buffer() which calls into random() three times per i.Also observe the numbers you posted, on ARM you get "sys 0m25.370s". This thing does no IO at all, no syscalls itself. What could cause this? Note: Locks on Linux typically boils down to futexes, whose slow (contended) path goes through the kernel (ouchies).Also I wouldn't be at all surprised if "while (i--) buffer[i ]= (random()-random())/(random() + 1.0);" issignificantlymore expensive than "while (i--) buffer[ i]= (((double*) arg)[ i] + buffer[i ]) / 2.0;". Depending on the PRNG used by the libc, quite some of these are very slow, much slower than 2 FLOP (add, div).---Ok. Let's do this.Your version (bench1.c).Code:#include <pthread.h>#include <stdlib.h>#include <stdio.h>#define BUF_SIZE 16*1024*1024pthread_t t1, t2, t3, t4;double* nu_buffer () {double* buffer= malloc(BUF_SIZE);int i= BUF_SIZE/8;while (i--) buffer= (random()-random())/(random() + 1.0);return buffer;}void* thread_proc (void* arg) {int i= BUF_SIZE/8;double* buffer= nu_buffer();while (i--) buffer= (((double*) arg) + buffer) / 2.0;return buffer;}void do_threads_stuff (double* buffer) {pthread_create(&t1, NULL, thread_proc, buffer);pthread_create(&t2, NULL, thread_proc, buffer);pthread_create(&t3, NULL, thread_proc, buffer);pthread_create(&t4, NULL, thread_proc, buffer);void** r;pthread_join(t1, r), free(*r);pthread_join(t2, r), free(*r);pthread_join(t3, r), free(*r);pthread_join(t4, r), free(*r);}int main (void) {double* buffer= nu_buffer();do_threads_stuff(buffer);//free(buffer); return 0;}Note corruption in do_threads_stuff. Let's fix that first, shall we?Code:#include <pthread.h>#include <stdlib.h>#include <stdio.h>#define BUF_SIZE 16*1024*1024pthread_t t1, t2, t3, t4;double* nu_buffer () {double* buffer= malloc(BUF_SIZE);int i= BUF_SIZE/8;while (i--) buffer= (random()-random())/(random() + 1.0);return buffer;}void* thread_proc (void* arg) {int i= BUF_SIZE/8;double* buffer= nu_buffer();while (i--) buffer= (((double*) arg) + buffer) / 2.0;return buffer;}void do_threads_stuff (double* buffer) {pthread_create(&t1, NULL, thread_proc, buffer);pthread_create(&t2, NULL, thread_proc, buffer);pthread_create(&t3, NULL, thread_proc, buffer);pthread_create(&t4, NULL, thread_proc, buffer);void* r;pthread_join(t1, &r), free(r);pthread_join(t2, &r), free(r);pthread_join(t3, &r), free(r);pthread_join(t4, &r), free(r);}int main (void) {double* buffer= nu_buffer();do_threads_stuff(buffer);//free(buffer); return 0;}Ok. Let's run that.Code:3.06user 1.63system 0:01.39elapsed 337%CPU (0avgtext+0avgdata 83264maxresident)k0inputs+0outputs (0major+643minor)pagefaults 0swapsOk, pretty much the same bad result you got. Note high system time.Now let's fix that random() mess.Code:#include <pthread.h>#include <stdlib.h>#include <stdio.h>#include <string.h>#define BUF_SIZE 16*1024*1024pthread_t t1, t2, t3, t4;double* nu_buffer () {double* buffer= (double*)malloc(BUF_SIZE);struct random_data rd;char sb;memset(&rd, 0, sizeof(struct random_data));initstate_r(random(), sb, 64, &rd);int i= BUF_SIZE/8;while (i--) {int r1, r2, r3;random_r(&rd, &r1);random_r(&rd, &r2);random_r(&rd, &r3);buffer= (r1-r2)/(r3 + 1.0);}return buffer;}void* thread_proc (void* arg) {int i= BUF_SIZE/8;double* buffer= nu_buffer();while (i--) buffer= (((double*) arg) + buffer) / 2.0;return buffer;}void do_threads_stuff (double* buffer) {pthread_create(&t1, NULL, thread_proc, buffer);pthread_create(&t2, NULL, thread_proc, buffer);pthread_create(&t3, NULL, thread_proc, buffer);pthread_create(&t4, NULL, thread_proc, buffer);void* r;pthread_join(t1, &r), free(r);pthread_join(t2, &r), free(r);pthread_join(t3, &r), free(r);pthread_join(t4, &r), free(r);}int main (void) {double* buffer= nu_buffer();do_threads_stuff(buffer);free(buffer); return 0;}=Code:0.16user 0.00system 0:00.06elapsed 258%CPU (0avgtext+0avgdata 83176maxresident)k0inputs+0outputs (0major+639minor)pagefaults 0swapsOn this machine the difference is even a bit closer to 20.

author: dom0
author_rank: Super Contributor
timestamp: April 30, 2017, 08:19:38 pm
content: Btw. I don't see what you're attempting here, it's not only a micro-benchmark but also an markedly bad one. There are much better vetted benchmarks for this. Heck, even ol' LINPACK is better at this. Try STREAM for a quick gloss over the memory subsystem. Take a look at lmbench3 for microbenchmarks that actually microbenchmark interesting properties. Say, lat_ctx.

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: April 30, 2017, 08:32:51 pm
content: I get:jorge@unibody:~/kk$ gcc -O0 threads.c -o threadsthreads.c: In function ‘nu_buffer’:threads.c:20: error: storage size of ‘rd’ isn’t knownline 20 is:struct random_data rd;

author: dom0
author_rank: Super Contributor
timestamp: April 30, 2017, 08:35:47 pm
content: This is a glibc extension.With other libc's you may have to use their equivalent or if they don't have one bring your own PRNG.

author: Marco
author_rank: Super Contributor
timestamp: April 30, 2017, 08:45:56 pm
content: Quote from: dom0 on April 30, 2017, 05:18:26 pmIMO the issue has been that as you add wider superscalar execution and other architectural changes with the same ISA, elegant ISAs with features purpose designed for a given implementation (VLIW, delay slots etc) only get in the way with the next generation of processor trying to wring out a few extra percent of IPC. Even the instruction bloat of RISC becomes a disadvantage. When you have to heap hacks and cludges on a giant pile to apply increasingly speculative execution with an instruction set and software never designed/compiled for it, CISC becomes an advantage.QuoteAFAICS they just got rid of the trace cache.

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: April 30, 2017, 08:47:05 pm
content: Quote from: dom0 on April 30, 2017, 08:17:04 pmSorry I dont get it, how isCode:void** r;pthread_join(t1, r), free(*r);pthread_join(t2, r), free(*r);pthread_join(t3, r), free(*r);pthread_join(t4, r), free(*r);Different than:Code:void* r;pthread_join(t1, &r), free(r);pthread_join(t2, &r), free(r);pthread_join(t3, &r), free(r);pthread_join(t4, &r), free(r);?Edit: Never mind

author: dom0
author_rank: Super Contributor
timestamp: April 30, 2017, 08:59:22 pm
content: Quote from: Marco on April 30, 2017, 08:45:56 pmYes, ISAs that directly tie to the layout and timing of execution units were never sustainable; and VLIW as a general concept only worked so-so for compilers.The change away from VLIW in AMDs GPUs is an interesting facet here.-Core and Netburst are very different designs overall. The pipelines of the current µarchs are still much more like Core. One could say the design principles just don't work out; high clocks are worthless if the CPU is inefficient. IBM with their POWER µarchs found this as well.

author: dom0
author_rank: Super Contributor
timestamp: April 30, 2017, 09:00:36 pm
content: Quote from: GeorgeOfTheJungle on April 30, 2017, 08:47:05 pmAbove: r is an uninitialised pointer on the stack (likely garbage) and you tell pthread_join to write to that uninitialized pointer (so a write goes somewhere garbage-y in the memory).Below: r is a variable on the stack (which is also a pointer), and pthread_join writes to that stack variable.

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: April 30, 2017, 09:30:53 pm
content: Cool, now the OPi (with dom0's code above) gives:pi@orangepi:~$ gcc -lpthread -O0 threads.c -o threadspi@orangepi:~$ time ./threadsreal 0m0.831suser 0m2.070ssys 0m0.230sBut OSX does not have random_r, any ideas?

author: brucehoult
author_rank: Super Contributor
timestamp: April 30, 2017, 09:34:22 pm
content: Quote from: NiHaoMike on April 30, 2017, 04:34:21 pmI haven't heard that, but I would *expect* it, given that1) Apple's been designing their own CPUs for a while, and had 64 bit since the iPhone 5s in 2013.2) iOS apps have by default since mid 2015 been uploaded to the AppStore as llvm bitcode NOT machine code, and Apple delivers 32 bit versions to 32 bit devices and 64 bit versions to 64 bit devices. I believe a developer can still turn off bitcode for iOS, but it is compulsory for WatchOS and TVOS.Thus Apple have the ability to use a 64 bit-only ARM any time they want to .. or even another CPU type entirely.

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: April 30, 2017, 09:49:49 pm
content: Ok so I simply won't fill the 16MB buffers with random(), just average whatever happens to be there (most likely zeros or garbage) as doubles in parallel in 4 threads:https://gist.github.com/xk/b8b2ff4ab1455237906a8b13e3f1f02fThe i7 Mac:unibodySierra:Desktop admin$ gcc -O0 threads.c -o threadsunibodySierra:Desktop admin$ time ./threadsreal 0m0.054suser 0m0.084ssys 0m0.073sAnd the OPi Zero:pi@orangepi:~$ gcc -lpthread -O0 threads.c -o threadspi@orangepi:~$ time ./threadsreal 0m0.166suser 0m0.440ssys 0m0.140sAnd the Mac now is 0,166/0,054= 3x times faster, and more than 100x as expensive.

author: brucehoult
author_rank: Super Contributor
timestamp: April 30, 2017, 09:54:53 pm
content: Quote from: dom0 on April 30, 2017, 05:11:36 pmAnd what was wrong?QuoteStrangely enough, I know this. I even listed the ISAs.QuoteStrangely enough, I know this. I even mentioned Cavium. I hadn't heard for sure that Apple has done this but I've been expecting it ever since they required apps to include 64 bit versions.QuoteStrangely enough, I know all this, and said as much.QuoteYou're making my point for me.QuoteStrangely enough, I know this, and said as much.Except GPUs do actually have branch instructions. Or, at least, the one I'm currently helping write the compiler for does. The branch instruction looks exactly like any other CPUs branch instruction. It sets the "next PC" for that thread to the branch target and masks the thread off. If/when the actual PC becomes equal to the desired PC in the future the thread is masked back on.You can also work with explicit thread masks in any general register. That's how short nested if/then/else is most efficiently done. But it can be done with branch instructions, same as usual.QuoteIt never works worse than a CPU with the same number of in-order cores as the CPU has warps. Which there can be more than 100 of these days. Sometimes it works up to 32x better than that.QuoteI tried the Cavium 96 core machine for building llvm (one of my primary workloads). It had 65% of the performance of the top EC2 c4.8xlarge machine for 30% of the price. Pretty good.It beat the next lower c4.4xlarge in absolute terms.

author: brucehoult
author_rank: Super Contributor
timestamp: April 30, 2017, 10:00:17 pm
content: Quote from: dom0 on April 30, 2017, 05:18:26 pmYou just told me, in the previous post: "Similarly if you look at a POWER or SPARC core the decoder takes a large part of the logic transistor budget; in almost all CPU cores -regardless of ARM/x86/POWER/SPARC- nowadays the decoder (the parts you can identify as one) is usually the largest logic part (even on the floor plan)."Please do try to choose one or the other.QuoteAs I said in my original post, when you have massively OOO (e.g. 300 instructions in flight, register rename up the wazoo) the decode doesn't matter.It also, however, means that you're using ten times the transistors and power to get twice the single core performance. This makes sense in desktop systems. It doesn't make sense in either phones or the datacentre.

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: April 30, 2017, 10:19:23 pm
content: Quote from: dom0 on April 30, 2017, 05:22:56 pmNote to the casual reader: that ^^^ is bad and renders the fp benchmark ~ useless (for the intended purpose). Other (hopefully better) tests show that the Intel i7 Mac was only 3x (not 12x) times faster than the $8 Orange Pi Zero in this silly average fp/doubles benchmark.

author: JuanAG
author_rank: Newbie
timestamp: May 01, 2017, 09:14:08 pm
content: Quote from: GeorgeOfTheJungle on April 30, 2017, 07:04:30 amMore transistors doesnt mean more performance (a 4 cilinder car can have more horsepower than a V8)1/3 or the transistors are for the graphics, 1/5 is for the L3 cache and if you see the core itself only a small amount of transistors are the real power in form of ALUsSo more transistors if they doesnt do anything useful doesnt mean nothing, a supercomputer is doing the same operations again and again, it focuses on few instructions and much of wich will have the same cycles in arm or intel so it performs same clock-to-clock comparation, only in "rare" operations like sqrt or similar exists a difference but arm is more flexible and it will offer more instructions to match better the code that x86 which doesnt have all the operations + increment so in x86 will be two cycles, the op and later the inc.Or the registers in wich case x86 has few so you has to store and load more from the caches, the pi cpu has 29 64bits (are 32 but three are reserved) Xserires, another 29 (32 really) 32 bits Wseries and 30 if i remember right (of 32) 128 bits Vseries (these are special and can be broken to 64 bits Dseries or 32 bits Sseries or 16 bits Hseries) meaning that you almost dont need to store because you run out of registers and load later as most as you has to do in x86The compiler is the key, i do asm my self so i can see first hand that optimized code for both performs the same and if you take time to optimize to an insane level arm will do betterARM is not only for low power applications, intel had to say goodbye to the atoms line because it performs worst drawing more power than an ARM cpu

author: Mechatrommer
author_rank: Super Contributor
timestamp: May 02, 2017, 12:46:31 am
content: isnt graphics/cache transistors are there for performance? try running altium designer, photoshop, autodesk inventor in arm, if arm can do as well as intel, i'll buy arm computer. everybody try to be an engineer here.

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: May 02, 2017, 07:50:44 am
content: This is very interesting: "What does an instruction set designer do when she is given moar transistors?"And this too:

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: May 02, 2017, 07:57:35 am
content: Quote from: JuanAG on May 01, 2017, 09:14:08 pmNo doubt if you do that you may find some sweet spots, but that's not how things work. Take a linux performance benchmark instead to test the system as a whole and you'll see what are those billions of transistors there for.E.g. try to fine tune the benchmarks above and make them run faster in the OPi, I doubt very much you'll find a sweet spot there because the caches are less clever and much smaller, the buses are narrower and much slower, the RAM as well, etc. IOW the ARM has less hardware features (=>less transistors =>less power) of the kind that make a CPU perform better.

author: dom0
author_rank: Super Contributor
timestamp: May 02, 2017, 08:58:18 am
content: Quote from: blueskull on May 02, 2017, 06:30:51 amThat's why there is no iGPU on server chips (sockets R and such).

author: cprobertson1
author_rank: Frequent Contributor
timestamp: May 02, 2017, 09:53:05 am
content: Crikey! I take my eyes off the thread for two minutes and it runs away without meFormyapplication, there doesn't appear to be too many FPOs - so a higher MIP rate would probably be advantageous - having said that - there are several nested layers of inefficiencies slapped on top of that, so I'd find it hard to characterise my code in terms of resources actually required - which was why I was fairly generic in my "just need clockspeed" requirement - which is vague and pretty misleading as far as these things goMy assumption here is that "once running it runs as fast as the processor allows" - so a faster processor ? faster computation (negative losses between RAM <-> CPU <-> HDD of course): so rather than building anoptimumsystem, I had figured a more generic system with a high overall clock speed would be adequate (obviously there are diminishing returns here: especially since my code is not optimised to eke the most out the available hardware).As I mentioned, I've tested this with a beowulf cluster of 4x Raspberry Pi - and I've just done a rough benchmark this weekend to figure out the effects of additional processing units.[% increase][% efficiency]12525002525000100249740024490097973484900240000989544728002328009792*Average tests run per hourThese ran for three hours on each setup and the average calculated (and rounded to the nearest 100) - as you can see there's an approximate 3% loss with each additional stage (there is a chaotic factor involved since it's a genetic algorithm - sometimes it gets lucky and the code is very effective against the opponent-code, in which case the test proceeds quickly; other times it gets unlucky and they sit there circling each other and resolves in a tie after a certain number of cycles.)Im not sure if there are additional bottlenecks in the network switch that will cause the losses to escalate further with increasing node count.I also suspect the control node may at some point struggle to generate sample code - though at the moment it takes 30 seconds to generate 8000 code samples (working out at 960'000 tests, which takes ~2 hours to perform under the beowulf cluster) so I don't think it's too stressed.I have also discovered that staggering the testing start times on the control nodes reduces the amount of data going back to the control node at any given time: which has a disproportionate effect on overall speed/efficiency (the slave nodes sit idle while waiting for orders - if the control node receives data from all the slave nodes simultaneously it deals with the first one and the rest get buffered until it's finished. The result is that each slave node has to wait for every other slave node before it - time that could be spent processing; by staggering them there is no buffering and no more than one slave is sitting idle at any particular time (unless one finishes late while the next one finishes early)Anyhoo, I'm going to read through all this again - quite a lot of interesting stuff here!

author: dom0
author_rank: Super Contributor
timestamp: May 02, 2017, 11:22:18 am
content: Hm, from a software perspective this sounds like an interesting project to write a (small) JIT for your bytecode (I assume). They're great fun to build

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: May 02, 2017, 12:10:49 pm
content: Quote from: cleaningOut on May 02, 2017, 01:19:45 amPorted to ARM? What does that mean? Most things these days are written in C or higher level languages, it should be a matter of hitting "compile" in the proper toolchain (yeah, I know, I know...).

author: brucehoult
author_rank: Super Contributor
timestamp: May 02, 2017, 12:26:05 pm
content: Quote from: GeorgeOfTheJungle on April 30, 2017, 09:30:53 pmOn an Odroid XU4 (quad Cortex A15 @2.0 GHz) I got:real 0m0.398suser 0m0.710ssys 0m0.300s

author: CJay
author_rank: Super Contributor
timestamp: May 02, 2017, 12:26:19 pm
content: Quote from: GeorgeOfTheJungle on May 02, 2017, 12:10:49 pmWould be interesting to see how a current Intel chip stacks up against something like an ARM, core to core so to speak if the compiler/hardware could be persuaded to run code on just the CPU core and not the 'extra' (potentially) performance enhancing on chip peripherals.An almost but not quite comparable example, ISTR the 1940s Bletchley Bombes were capable of outpacing a generic desktop PC CPU until a few years ago, obviously that was a highly specialised application and one that the Bombes had been specifically designed and optimised for but even so, 60 or so years of technological development hadn't produced a generic machine that was any faster

author: brucehoult
author_rank: Super Contributor
timestamp: May 02, 2017, 12:34:33 pm
content: Quote from: GeorgeOfTheJungle on April 30, 2017, 09:49:49 pmAnd for this, Odroid XU4 is:real 0m0.212suser 0m0.130ssys 0m0.625sAnd Odroid c2 (quad A53 @1.5 GHz):real 0m0.142suser 0m0.350ssys 0m0.140sThis benchmark is really too short to measure accurately on any of these systems, but especially Intel.Both the Odroids have less User time than the Orange Pi, the XU4 by a factor of 3.4x. They are both Raspberry Pi-sized SBCs. The C2 costs $5 more than a Pi. The XU4 is about twice the C2, but a 4A power supply is included. Both have 2 GB RAM, gig ethernet, and much faster SD card and USB than a Pi (and eMMC too).

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: May 02, 2017, 12:50:30 pm
content: Bruce I have to admit that I swithed the OPi to the performance governor for the tests:Code:cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governorinteractiveecho performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governorcat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governorperformanceBecause if you don't do that it takes a while for the cpu to speed up...

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: May 02, 2017, 02:33:43 pm
content: But the xu4 has 8 real cores, so let's try that versus the i7 4 cores + hyperthreading shall we?8 threads and 96MB buffers :https://gist.github.com/xk/ba76a426e4c391c0b22b3eddbdb11898The xu4 8 ARM cores big/little:pi@xu4:~$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governorperformancepi@xu4:~$ gcc -lpthread -O0 threads.c -o threadspi@xu4:~$ time ./threadsreal 0m2.502suser 0m6.875ssys 0m9.675sThe Intel i7 4 cores+hyperthreading:MBP-17:Desktop admin$ gcc -O0 threads.c -o threadsMBP-17:Desktop admin$ time ./threadsreal 0m0.383suser 0m0.904ssys 0m0.874s

author: brucehoult
author_rank: Super Contributor
timestamp: May 02, 2017, 04:05:20 pm
content: Quote from: GeorgeOfTheJungle on May 02, 2017, 02:33:43 pmNot really. The big and LITTLE cores don't ever run at the same time as each other. For performance questions you should ignore the LITTLE cores.

author: GeorgeOfTheJungle
author_rank: Super Contributor
timestamp: May 02, 2017, 04:12:07 pm
content: Quote from: brucehoult on May 02, 2017, 04:05:20 pmI see the 8 cores @ 100% there in htop:Code:pi@xu4:~$ cat loop.shwhile [ true ]; do./threadsdonepi@xu4:~$ sh loop.sh