Very small linux capable core - Page 1
Forum section: electronics / general

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: July 25, 2021, 01:48:57 pm
content: Hi,This could be my first time going for a linux on a soft-core, I want to know if we have open source cores capable of running minimal linux, I have about 10-12K of LUT left on my device, and it's a Gowin part.Do you recommend any thing? is it possible on 10K LUTs?

author: brucehoult
author_rank: Super Contributor
timestamp: July 25, 2021, 01:56:04 pm
content: Sure.Quotehttps://github.com/SpinalHDL/VexRiscv

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: July 25, 2021, 01:59:27 pm
content: Thanks for the info,It's in scalado you recommend anything Verilog or system verilog at least! or do you recommend any good tutorial about scala?How long does it take to learn scala, since I can do a moderate job on verilog and system verilog.

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 25, 2021, 04:10:12 pm
content: Quote from: ali_asadzadeh on July 25, 2021, 01:59:27 pmIt uses SpinalHDL. It's based on Scala so sure, you'll need to learn Scala. And then you can go there:https://spinalhdl.github.io/SpinalDoc-RTD/But SpinalHDL generates Verilog, so you can read the generated code, although I would expect it not to be extremely readable compared to human-written Verilog.Also, although probably more comfortable, you don't need to understand VexRiscv fully to be able to use it. Many people have used it without knowing SpinalHDL. You just need to figure out how to configure it for your particular use and generate the HDL output.Now, it's unclear to me whether VexRiscv is really ready to be directly used for running Linux at the moment outside of pure simulation. Quoting VexRiscv's Readme:QuoteMaybe brucehoult can confirm it is possible already, but what the maintainers say sounds confusing to me.

author: brucehoult
author_rank: Super Contributor
timestamp: July 26, 2021, 12:40:48 am
content: Quote from: SiliconWizard on July 25, 2021, 04:10:12 pmI don't have direct experience with it.The safe choice for SoC is RocketChip. I don't know the minimum size if you cut out FP and so forth. A full single-core system will fit on an Arty. Several real chips are based on the Rocket components, including the SiFive Fe-310 and FU-540, and Kendryte K210.Again, it's a generator (Chisel this time) written in Scala, producing verilog.

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 26, 2021, 02:01:28 am
content: But then I would doubt a minimal RocketChip SoC able to run Linux would fit witing 10k LUTs?

author: brucehoult
author_rank: Super Contributor
timestamp: July 26, 2021, 03:16:29 am
content: Quote from: SiliconWizard on July 26, 2021, 02:01:28 amThe core itself, very easily.I just found this:https://groups.google.com/a/groups.riscv.org/g/hw-dev/c/zZxy0iFzrvI/m/LVeFiK2vAQAJRocketTile (which includes L1 caches) 4413 LUTs.11000 LUTs in total for a system, but that's mostly in various bus interfaces. 11000 is not *that* far off 10000.

author: asmi
author_rank: Super Contributor
timestamp: July 26, 2021, 03:52:40 am
content: I think this is a bit of an academic discussion, because Linux requires a lot of memory (relatively speaking), which in turn requires external memory in all but the largest FPGAs (for which there is no point in saving LUTs in the first place because you can fit dozens of even the largest cores). So your SoC will have to have some sort of bus interface and external memory controller. To give one data point, Petalinux (embedded Linux build which runs on Microblaze CPU) requires at least 32 MBytes of RAM. When I experimented with Petalinux on my Spartan-7 board, OS ended up consuming about 20 MBytes of RAM for itself, so for 32 MB of total RAM that doesn't leave very much for user applications. I don't know what RISC-V build of Linux requires (and I'm really curious to find out, so if someone knows - please let us know), but something tells me the requirement is in the same order of magnitude.

author: brucehoult
author_rank: Super Contributor
timestamp: July 26, 2021, 04:27:11 am
content: Quote from: asmi on July 26, 2021, 03:52:40 amSure. The assumption here is always going to be that you have external RAM of some kind -- DRAM with the FPGA vendor's DDR IP (not counted in the 10k LUT budget), or you can get 32Mx16 PSRAM for $7 that can be interfaced yourself very easily (i.e. 2 chips, 32Mx32, 128 MB).RISC-V mainline Linux is very similar I've played around with a javascript implementation with Fedora here:https://bellard.org/jslinux/vm.html?cpu=riscv64&url=fedora33-riscv.cfg&mem=3232 MB lets you boot into console with 15 MB free (19 available if freeing disk cache). You can use gcc to compile a small program, vi to edit it etc. It still works with 24 MB or even 20 MB.Buildroot with 16 MB gives you 5 MB free, and is enough to let you compile and run hello.c. It even works in 15 MB, but that's the limit if you want to run gcc. If you just want to run existing undemanding binaries then 12 MB might be enough.https://bellard.org/jslinux/vm.html?cpu=riscv64&url=buildroot-riscv64.cfg&mem=16You might be able to trim Buildroot to run in 8 MB, after some fashion.Of course Fedora requires RV64GC, so double precision FPU etc, which is big. Buildroot or yocto/OpenEmbedded you could configure for RV32, no FPU -- not even integer multiply/divide if you wanted to.

author: asmi
author_rank: Super Contributor
timestamp: July 26, 2021, 04:45:10 am
content: Quote from: brucehoult on July 26, 2021, 04:27:11 amPSRAMs tend to have big access latencies, and thus require relatively large caches, contributing to the overall resource demands. For example, HyperRAM devices have latencies in the order of tens of cycles, as they are optimized for streaming data (and so sequential access), so cache lines better be rather large to amortize such big latency.Also, I don't know much about Gowin, but from what I do know they are kind of similar to ice40Ultra series of FPGA, and so are VERY pin- and resource constrained. And memory interfaces typically consume a lot of pins. While doing such a project could be an interesting challenge, I question practicality of such approach. Especially since the point of using Linux is typically to use some of it's software stacks like TCP/IP or USB Host, both of which require controllers which further raise the resource requirements (especially the latter one). If you don't need any of those, you are much better off going for bare-metal application.

author: brucehoult
author_rank: Super Contributor
timestamp: July 26, 2021, 05:24:31 am
content: Quote from: asmi on July 26, 2021, 04:45:10 amFPGA soft cores aren't all that fast either -- probably not more than 30 or 50 MIPS on a Gowin.The PSRAM I referred to...https://www.mouser.com/datasheet/2/1127/APM_PSRAM_OPI_Xccela_APS256XXN_OBRx_v1_0_PKG-1954780.pdf... has oct SPI at 200 MHz for 800 MBps burst of 16, 32, or 64 bytes (or 2K). With SPI it's not a lot of pins either.That seems like way more bandwidth than a soft core CPU could use.QuoteI don't disagree with that, I'm just trying to answer the OP's question.

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: July 26, 2021, 06:35:34 am
content: Thanks for sharing your Ideas.In the gowin I have about 12K of LUT's, also I have a 32MB internal SDRAM, I want to compile and use thishttps://libiec61850.com/libiec61850/on minimal linux, I hope I can do it.

author: brucehoult
author_rank: Super Contributor
timestamp: July 26, 2021, 07:17:12 am
content: Quote from: ali_asadzadeh on July 26, 2021, 06:35:34 amSo now I don't know why you need Linux.It should be easy to create a HAL for FreeTOS or Zephyr for that project.

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: July 26, 2021, 07:44:33 am
content: QuoteI think this project requires some libraries that needs a minimal linux.

author: brucehoult
author_rank: Super Contributor
timestamp: July 26, 2021, 07:55:53 am
content: Quote from: ali_asadzadeh on July 26, 2021, 07:44:33 amAccording to the link you sent, it needs only time, threads, TCP/IP sockets, ethernet, and a filesystem. The mentioned RTOSes have all those.

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: July 26, 2021, 08:22:25 am
content: So, what soft processor do you suggest for that?

author: brucehoult
author_rank: Super Contributor
timestamp: July 26, 2021, 08:36:31 am
content: Quote from: ali_asadzadeh on July 26, 2021, 08:22:25 amThe same as already mentioned, but you could use a smaller configuration without MMU etc. Does that library use floating point? I didn't look.

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: July 26, 2021, 08:49:03 am
content: It can use floating point.

author: brucehoult
author_rank: Super Contributor
timestamp: July 26, 2021, 09:18:10 am
content: Quote from: ali_asadzadeh on July 26, 2021, 08:49:03 amA decent single precision FPU will use 2000 LUT6s all by itself, or around 4000 for double precision. More with LUT4s, obviously.

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 26, 2021, 04:35:52 pm
content: Quote from: asmi on July 26, 2021, 03:52:40 amAgreed. This is why I talked about a "real" SoC able to run Linux (with the idea of doing anything useful with it.)And if you want to support DDR RAM, then the controller will itself take several more thousands of LUTs. If SDRAM is OK (but then you'll probably have very limited memory), then it becomes doable.What I'm guessing here is the OP would like to get an alternative to using a typical COTS SoC supporting Linux to lower cost/part count, while having access to FPGA fabric for some custom logic.But they would need to also tell us what it is they would intend on running on this.The small RISCV cores, such as VexRiscv or PicoRV32, usually have relatively poor performance, with average CPI in the order of 4 or so. With no FPU. So if you are lucky to get them to clock at up to around 150 MHz (or even 200 MHz) on a modest FPGA, then it would probably have the equivalent processing power of an old ARM CPU @ 40 MHz or something (or even slower). Are you sure you really want to run Linux on this? What would that be for exactly?

author: asmi
author_rank: Super Contributor
timestamp: July 26, 2021, 05:25:36 pm
content: Quote from: SiliconWizard on July 26, 2021, 04:35:52 pmSDRAM is good enough if only used for the CPU as there won't be much of a bandwidth left to to anything useful.Quote from: SiliconWizard on July 26, 2021, 04:35:52 pmOP doesn't have a problem with using shady Chinese parts, so I'm sure if that were the case, he'd be able to source some cheap-ass ripoff parts. I've seen some Zynq-010's on Aliexpress for a pocket change, and it's got more resources than Gowin parts + two A7 cores.Quote from: SiliconWizard on July 26, 2021, 04:35:52 pmJust for fun, I'm currently working on an out of order core (Tomasulo with ROB) - curious to see what kind of performance it can reach with that approach. Especially curious at the kind of frequency it can achieve on a K325 SG2 on a Genesys 2 board.

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 26, 2021, 06:13:29 pm
content: Quote from: asmi on July 26, 2021, 05:25:36 pmI'm curious too. Getting it right and 100% bug-free is no small endeavour. Will that be a fully-pipelined core?

author: asmi
author_rank: Super Contributor
timestamp: July 26, 2021, 06:39:16 pm
content: Quote from: SiliconWizard on July 26, 2021, 06:13:29 pmOf course - that's the whole point of OoO execution. There are a couple of major difficulties with implementing such approach on FPGA, mostly with multi-ported memories for the reservation stations, ROB and register file. That's why it's an interesting challenge. RSes can probably be implemented in logic (as there are usually just a handful of them), but ROB and regfile are going to be interesting. Especially if multiple execution units will be able to "commit" results into regfile and ROB in a single cycle (think about the case when load and ALU operation complete in the same cycle, or even both of those + a INT MUL unit), or these things will need to be serialized (which will decrease performance). We will see.The two big advantages of Tomasulo-based approach is that you get register renaming effectively for free, and different operations can have different latencies, which eliminates the need for bypassing.

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 26, 2021, 07:38:31 pm
content: Good luck with that. =)I'm curious to see what kind of performance you get with this. I'll be curious about the Coremark score you're able to achieve, for instance, or the results you get with Bruce's countPrimes().

author: brucehoult
author_rank: Super Contributor
timestamp: July 27, 2021, 12:52:45 am
content: Quote from: asmi on July 26, 2021, 05:25:36 pmVery cool!Did you know that cpu designer Mitch Alsup  has recently come up with an enhanced CDC 6600 scoreboard algorithm that he says is equivalent to Tomasulo but easier to implement? His own description of this is not easy to access, but there is a detailed write-up of it by someone else here:https://libre-soc.org/3d_gpu/architecture/6600scoreboard/Please note: I endorse Alsup, I do not endorse the author of this page who may well have introduced some distortions or misunderstandings. co-author of the important 1991 paper "Single instruction stream parallelism is greater than two" that prompted modern OoO design. "When all constraints are removed except those required by the semantics of the program, we have found degrees of parallelism [in SPEC] in excess of 17 instructions per cycle. Finally, and perhaps most important for exploiting single instruction stream parallelism now, we show that if the hardware is properly balanced, one can sustain from 2.0 to 5.8 instructions per cycle on a processor that is reasonable to design today". Mitch designed Motorola CPUs in the 80s including the 88k. He was Chief Architect at AMD in the period when Athlon64/Opteron was being developed.

author: asmi
author_rank: Super Contributor
timestamp: July 27, 2021, 02:53:16 pm
content: Quote from: brucehoult on July 27, 2021, 12:52:45 amThanks! I skimmed through the link, and immediately noticed it's talking about 5R4W reg file - which was likeMulti-ported memory is the biggest problem in FPGA as BRAMs are only dual-port. More read ports can be added by adding duplicate BRAMs and writing to all of them at once, but this won't work if more than a single write port is required. If you serialize writing to reg file from the execution units, you can get away with only a single write port, but in any case you will need a metric ton of read ports (two per computational EU, one for a load unit, two for store unit), or a complex forwarding matrix (a-la CDB in the classic Tomasulo). So such core will be a BIG resource hog, which is why I'm curious to see if advantages of OoO with speculative execution are going to be worth it.I already have a RV64I core built as a "classic" pipeline running at a bit over 170 MHz on my Spartan-7 board, so it will be interesting to compare them.

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 27, 2021, 04:06:48 pm
content: Quote from: asmi on July 27, 2021, 02:53:16 pmHave you "benchmarked" it in some way? I'm curious.

author: asmi
author_rank: Super Contributor
timestamp: July 27, 2021, 06:30:34 pm
content: Quote from: SiliconWizard on July 27, 2021, 04:06:48 pmNot really, it's a bit of a mess and could use some polishing (like adding dynamic branch prediction), but I plan to add it because I want to add it to the new core as well. I want to try an OoO approach to see if it's worth going with that approach, or for FPGA it's better to stick to the classic pipeline, before I invest more significant amount of time into further development. BTW - that same core closes timing at 258 MHz on K325-2 of G2 board, which is a quite nice 50% frequency bump.

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 27, 2021, 06:54:44 pm
content: Quote from: asmi on July 27, 2021, 06:30:34 pmWell, if you haven't benchmarked what you already have, I can suppose it's not really ready for running any significant amount of code, in which case it's hard to tell how it will really perform for real use. Or not sure what you mean by "mess" and why you haven't tried some benchmarks just to see where you stand.I'm not sure if going for something a lot more advanced makes much sense if you haven't completed the simpler design first, but that's just me.Fmax is cool but tells you nothing about performance, and the whole point of OoO execution is perfomance. So yeah... I dunno. Just a thought, not to deter you. Just that I have been working on a RISCV core myself and I know it took quite a while to get it fully debugged, with enough around the core (some peripherals, memory cache, etc) to get it to run "significant" code. So my experience just tells me that going for a variant with OoO before everything else was finalized wouldn't have been the best idea. Just a thought, of course I absolutely do not know what your own goals are and I don't mean to lecture you.

author: asmi
author_rank: Super Contributor
timestamp: July 27, 2021, 08:35:33 pm
content: Quote from: SiliconWizard on July 27, 2021, 06:54:44 pmThere is a difference between validation and performance benchmark. I've done the former, but not the latter. That means the core executes code correctly and so it can successfully run any (valid) code.I will see if I can run brucehoult's code when I have some spare time.

author: brucehoult
author_rank: Super Contributor
timestamp: July 28, 2021, 12:22:29 am
content: Quote from: asmi on July 27, 2021, 02:53:16 pmThat's extremely fast for an FPGA core if it's using only 1 CPI (for non-load/branch). I'd be tempted to say "impossible" in an Artix-7. Is Spartan-7 *that* much better? I don't have experience with them.

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 28, 2021, 12:30:15 am
content: Quote from: brucehoult on July 28, 2021, 12:22:29 amI doubt it can be anywhere close to 1 CPI, especially for a 64-bit core, but even for a 32-bit one, at this clock rate. Or if it is, tell us your secret.Spartan-7 are stripped-down Artix-7 AFAIK.

author: brucehoult
author_rank: Super Contributor
timestamp: July 28, 2021, 12:52:38 am
content: Quote from: asmi on July 27, 2021, 06:30:34 pmDual-issue in-order with both an "early" ALU stage and "late" ALU stage is far far simpler, less area, less energy than OoO but gets much of the performance benefit, at least on code that runs in L1 cache / SRAM.Examples of this setup include the ARM A55, SiFive U74, and the Western Digital SweRV.On my primes benchmark (https://hoult.org/primes.txt) the dual-issue in-order SiFive U74 in the HiFive Unmatched uses almost exactly the same number of clock cycles as the 3-issue 128 entry ROB OoO ARM A15 in the Beagle-X15 and Odroid XU4. I think the difference (including between X15 and XU4) comes down to different gcc versions.The A57 is also 3 issue 128 entry ROB, and I think is pretty much just an Armv8 version of the A15, so is probably the same, but I don't have one to test.The A72 in a Pi 4 uses 17.5% (A32) to 19.7% (A64) fewer clock cycles than the above.In comparison, going from the single-issue U54 to the dual-issue U74 results in a 42.5% decrease in clock cycles.

author: asmi
author_rank: Super Contributor
timestamp: July 28, 2021, 01:06:01 am
content: Quote from: brucehoult on July 28, 2021, 12:22:29 amSpartan-7 and Artix-7 have the same fabric. The only difference is that there is a speed grade 3 for Artix, but not for Spartan, so technically Artix can be a bit faster. All operations are single-cycle and fully pipelined, but of course there are stalls due to branches/jumps, as well as data hazards because the core is only partially bypassed. Like I said above, I plan to eventually add dynamic branch prediction, as well as more bypassing. Also currently interrupts and exceptions are not implemented, however I don't think it's going to be that hard to add them.

author: asmi
author_rank: Super Contributor
timestamp: July 28, 2021, 01:13:28 am
content: Quote from: SiliconWizard on July 28, 2021, 12:30:15 amThere is no secret. If you throw enough hardware at the problem, you can achieve a lot. My core currently consumes 3424 LUTs and 1525 FFs, and I use post-route physical optimization to help close timing. See attachments for details.

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 28, 2021, 01:30:17 am
content: Alright. Of course it was all in what was meant by "1 CPI".Your core is not fully bypassed as you said (and has no branch prediction). I'm curious, as I said, to see real performance in terms of average CPI on real code, because it's likely to be pretty far from 1. The fact most instructions independently only require one cycle is the best case, and I don't doubt you can reach 170 MHz for that.I've done extensive simulations - and then real tests on FPGA - with various configurations, so I kind of know what to expect by now. You can have most instructions (except branches) individually 1 CPI, but the average CPI on any real code with no or little bypassing and no branch prediction is likely to be something between 2 and 3. So, I'm curious.And, I don't claim my implementation is optimized in terms of Fmax either - but I do know cores with similar features rarely get above 100 MHz or so on this kind of FPGAs. That's probably also why Bruce was surprised.

author: asmi
author_rank: Super Contributor
timestamp: July 28, 2021, 02:23:18 am
content: Quote from: SiliconWizard on July 28, 2021, 01:30:17 amIn my case bypassing shouldn't change much in terms of frequency, because it's not on a critical path, as I have a dedicated stage (read registers) which does just that - read registers and pass values along to the EX stage. One bypassing which IS on a critical path (from EX/MEM to RR/EX) is already there and implemented - this way I only have 2:1 MUX in front of ALU (which is on a critical path and limits Fmax now, specifically shifter) - it either takes value from the EX/MEM pipeline register (effectively ALU output), or from register passed along from RR stage, based on a flag passed along from RR stage. Another trick - I have separate ALUs for 64bit and 32bit operations. If I could find a way to speed up 64bit shifter, I can increase Fmax even further.Same thing can be said about branch prediction - fetch stage is very simple - the only complication there is dealing with the fact that a memory bus has a width of 64 bits, while commands are 32 bit, and so it has to split 64 values into two commands, and it also has to deal with jump targets not aligned onto 64 bit boundary.

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: July 28, 2021, 06:23:28 am
content: asmi can the core be shared

author: hve
author_rank: Regular Contributor
timestamp: July 28, 2021, 09:01:50 am
content: Not sure if it is already mentioned:Assuming you have external memory attached..https://github.com/litex-hub/linux-on-litex-vexriscv

author: asmi
author_rank: Super Contributor
timestamp: July 28, 2021, 12:55:30 pm
content: Quote from: ali_asadzadeh on July 28, 2021, 06:23:28 amIt will be open source, but right now it's not ready for the prime time yet. Also it's optimized for frequency and performance, and not resource utilization.

author: SiliconWizard
author_rank: Super Contributor
timestamp: July 28, 2021, 04:27:10 pm
content: Quote from: ali_asadzadeh on July 26, 2021, 06:35:34 amAs can be read on the website, this library can be ported for any custom HAL, so you don't need Linux.https://libiec61850.com/libiec61850/documentation/You could probably use it on a baremetal system writing your own HAL support. Not requiring to run Linux will open many more options for what you want to achieve. Just a thought. You may have other reasons for running on Linux, but we don't know.

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: July 28, 2021, 05:51:11 pm
content: QuoteThat's greatGuys what's your ideas about Latticemico32 and zipcpu?I think at least the zipcpu blog is just a gold mine

author: brucehoult
author_rank: Super Contributor
timestamp: July 28, 2021, 08:53:44 pm
content: Quote from: SiliconWizard on July 28, 2021, 04:27:10 pmYeah, I already covered that in:https://www.eevblog.com/forum/fpga/very-small-linux-capable-core/msg3614765/#msg3614765Quotehttps://www.eevblog.com/forum/fpga/very-small-linux-capable-core/msg3614778/#msg3614778Quote

author: dolbeau
author_rank: Regular Contributor
timestamp: August 04, 2021, 12:41:21 pm
content: Quote from: SiliconWizard on July 25, 2021, 04:10:12 pmVexRiscv will run Linux in FPGA (not yet ASIC that I know of) with no issue, both in the SpinalHDL Saxon SoC and in the Migen Litex SoC. For this second case, the entire process is scripted inLinux-on-Litex-VexRiscv. Complete with SMP support, optional C, and optional FPU (can be shared by 1 to 4+ cores). The Litex SoC might require sufficient memory (at least 32 MiB, preferably more) and support optional ethernet, micro-sd, framebuffer, USB OHCI host, and you may even get SATA if you feel adventurous.I run a 4 cores RV32GCBK VexRiscv Litex SoC on a Qmtech Wukong board (Artix-7 100T), it works fine, including recompiling enough X11 by itself to run Doom.

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 04, 2021, 03:56:16 pm
content: Quote from: dolbeau on August 04, 2021, 12:41:21 pmOK, thanks for the feedback. At what frequency do the cores run?

author: dolbeau
author_rank: Regular Contributor
timestamp: August 04, 2021, 06:08:38 pm
content: Quote from: SiliconWizard on August 04, 2021, 03:56:16 pm100 MHz with no issue on the speed grade 2 Artix-7. It's mostly limited by the decode path I think (there's a tuning in VexRiscv to relieve some pressure related to C which is enabled, and all the B and K instructions add a lot of stuff to the decoder).IIRC, the 'vanilla' VexRiscv in default config (RV32IMA) reached at least 120 MHz on that board, I don't think I tried to push beyond that.

author: asmi
author_rank: Super Contributor
timestamp: August 04, 2021, 08:02:22 pm
content: Quote from: dolbeau on August 04, 2021, 06:08:38 pmYep, this is why in my core I've split decoding and register reading into separate stages. Otherwise it was on the critical path. Now execute/alu64 is on the critical path, and there is nothing I can do about it, and I don't want to make it multicycle. That is part of the reason why I want to try Tomasulo out of order approach - this way I can make shifts (which is the critical path inside alu) multicycle without affecting other ALU operations. I've also added full bypassing, and as I expected, it did not cause any timing problems.

author: brucehoult
author_rank: Super Contributor
timestamp: August 05, 2021, 12:59:20 am
content: Quote from: asmi on August 04, 2021, 08:02:22 pmYou can of course easily split the six layers of a logarithmic 64 bit shifter into two pipeline stages with three shift layers in each stage.The problem is this almost certainly isn't going to let you double the clock frequency as something else will become the critical path long before that. That means that while every other instruction will now go a little faster, shifts will now be slower. If the clock speed improvement is small (under 5%?) it might not be a win overall. Shifts are pretty common in some types of code, and using the base ISA (no B extension) there is a regrettable amount of back-to-back shifts to do things such as zero-extending a 32 bit unsigned value to a 64 bit value whenever someone has foolishly used an unsigned int (32 bits) as a loop index (etc) and then uses it to index an array. People who use size_t or similar for their loop indexes are fine. As are people who use "int".Coremark is an unfortunate offender here. They have gone out of their way to typedef critical variables as "unsigned int" because this produces better code than "int" on 64 bit ARM CPUs. But it de-optimises 64 bit RISC-V. For some time, RISC-V people were changing the typedef, but the benchmark owner made a ruling this is not permitted.The RISC-V B extension adds some instructions which do 64 bit computations after implicitly zero-extending one of the operands, which eliminates this problem. And also instructions to sign-extend and zero-extend 8, 16, and 32 bit quantities to 64 bits with a single instruction instead of a pair of shifts.But anyway -- it would be very interesting to know what becomes the critical path after you split shifts into 2 stages, and how much frequency increase you can then get.

author: asmi
author_rank: Super Contributor
timestamp: August 05, 2021, 02:27:53 am
content: Quote from: brucehoult on August 05, 2021, 12:59:20 amI think jump/branch handling is the next one on the line, but I'm not sure about it yet. Currently I have too much latency on the front end (PC to fetch), so that's the area I need to focus on, as well as dynamic branch prediction to reduce this latency. There were some design decisions made in this area, which I question now, so will see about that. I also plan to eventually add support for external memory bus, and this will basically require implementing caches.When I started this design, my target was to hit 150 MHz and some reasonable CPI (anything below 2 I consider acceptable). Right now it's about 4 running a very branch-heavy code (loop for a few million cycles, toggle GPIO, repeat), but it's also an ideal target for branch prediction to drastically improve CPI. There are also some memory operations, but there is little I can do about speeding them up. For timing reasons I had to add additional register after BRAM output to close timing, so the data memory latency is currently 2. Good thing is that I still have a healthy amount of headroom as far as frequency goes. Another good thing is that addition of full bypassing indeed brought CPI of compute-heavy linear code close to 1, so that once I add branch prediction and reduce instruction fetch latency, I can get a very good overall CPI. Right now I only have an early unconditional jumps detection, so it's possible to hand-optimize the code with branches to achieve very good CPI.

author: brucehoult
author_rank: Super Contributor
timestamp: August 05, 2021, 03:25:12 am
content: SRAM/cache latency of 2 is very common in RISC-V pipelines, for example all the Rocket-based cores are e.g. SiFive's original 3- and 5-series. In fact they are 3 cycles for sub-word loads. But they don't stall unless the value is actually used by the immediately following instruction.

author: asmi
author_rank: Super Contributor
timestamp: August 05, 2021, 04:59:37 am
content: Quote from: brucehoult on August 05, 2021, 03:25:12 amIn my case there are up to 2 stall cycles for memory access as the request is sent to the data memory after execute stage, so the value becomes available via forwarding only 2 cycles later, and perhaps even later in case read requests go on the AXI4 bus to a peripheral. Right now CPU doesn't stall for AXI4 writes as AXI4 specification guarantees transaction order and so I assume that writes will complete before any subsequent reads. This will probably need to be revised later (definitely so if I ever get to implementing paged memory), but for now it's good enough.

author: asmi
author_rank: Super Contributor
timestamp: August 05, 2021, 05:56:34 am
content: Hmm... I rewrote the blinker code in assembly and optimized branches (such that branch is not taken most of the time, followed by unconditional jump), and CPI jumped up to ~2.6. Fetch latency is still the limiting factor, so it looks like fixing it will need to be a priority.

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 05, 2021, 04:39:54 pm
content: Quote from: brucehoult on August 05, 2021, 03:25:12 amAre you sure about the last sentence? Even with a latency of 1, a stall is required in this case. So with a latency of 2, I think a stall is also required if the instruction after the immediately following instruction uses the value. The higher the latency, the more load-use hazard cases you get. Did I miss anything?

author: brucehoult
author_rank: Super Contributor
timestamp: August 05, 2021, 11:18:50 pm
content: Perhaps a terminology difference?I call the fastest instructions such as add, move etc latency 1. Perhaps you are calling them latency 0?

author: asmi
author_rank: Super Contributor
timestamp: August 06, 2021, 12:40:23 am
content: Quote from: brucehoult on August 05, 2021, 11:18:50 pmI think in this case a data memory latency is a measure of how many stall cycles can there potentially occur due to load-to-use data dependency. So no stall cycles - data memory latency is zero (as the result of previous operation is available to the following ones as soon as the execute stage for a previous command is completed), one stall - the following command will need to wait max 1 cycle - this is a "classic" implementation of 5-stage pipeline when memory operation begins as soon as execute is completed (and memory address is registered in EX/MEM pipeline registers), and completes one cycle later (so that you can forward it to subsequent commands). In my case I have a data memory latency of 2 cycles because memory operation only completes two cycles after request was made, and so the earliest that a value will be available to subsequent commands is 2 cycles after execution of load command.This latency can be masked by scheduling a load command two cycles before the value is used, or if your CPU can execute commands out-of-order (perhaps speculatively if branch is involved), and it looks like GCC does that reordering to a certain degree with -O3 optimization turned on. And of course as soon as you add support for external (to the core) memory-mapped resources (be it external RAM, or registers of peripherals), that constant latency stuff goes out the window, and you WILL get stalls due to cache misses, some sort of lags in peripherals, or a congestion at the memory controller, and only ways to somewhat hide them is to go for multi-issue and out-of-order execution with register renaming, as well as nonblocking cache such that other instructions can continue executing while cache miss is being handled.