Weird effect on Cyclone V + HPS after soft or hard boot - Page 1
Forum section: electronics / general

author: betocool
author_rank: Regular Contributor
timestamp: August 29, 2023, 01:35:45 pm
content: Hey all,I'm posting here because I'm experiencing something really funky on a SoC (Cyclone V HPS + FPGA) running Linux on a DE10 Nano board. Let me say first that this project is not terribly complex, but we are offloading some processing bits and pieces off to the FPGA, which pretty much freed up a complete ARM core and manages the task with flying colours. I've done a lot of simulation where I could (except the HPS and OS) and things are consistent. Timing analysis only shows unconstrained inputs/outputs.Here's the thing. There is a process, basically a histogram process, reads about 600Kbytes from memory and sorts the data in 1024 bins. So far so good. This process was moved from Python (numpy) to the FPGA because it was using 50% of the HPS, basically one processor. This data gets refreshed regularly, so we're talking a few MBytes per second.The approach to use the FPGA is:- Let the OS copy the incoming data to a specific memory space and specify the length- Signal the FPGA to start processing by setting a register start bit- Read a register every couple of ms to wait for the ready bit to be set.The above works. The data is copied by the OS and the FPGA uses the SDRAM FPGA-to-HPS interface, taking care of wait times and such. I know for a fact the data processed is correct because I have a checksum that I can check against, and I can compare both numpy and FPGA processes, the result is the same. The SDRAM FPGA-to-HPS is configures as read/write, even though the FPGA only reads from it.So far, so good. Now the issue.If I boot the system by taking power away, the process doesn't start after calling the script, not only that, it freezes the whole Linux OS without warning. SSH is lost, blinkys are lost, but two blinkys generated by the FPGA fabric still work. A power reboot later, same effect. No matter how often I power-boot, the whole OS crashes mercilessly after setting the "enable" bit.Now, if I do a power boot and then I do a soft boot using the "reboot" command (so many boots!), the process starts and finishes and runs without issues. I have only tested it a few minutes at a time, long term tests will come later. I can "reboot" as often as I want and the script starts after setting the enable bit. To be clear, I do not run or set anything between hard power-on and reboot.I checked the SDRAM register settings for before and after, they are the same. I'm guessing there must be something in the FPGA code that is initialised wrong, or tries to write something in memory, and kills it? But then after a reboot all is well?If you've experienced something like that and remember more or less what you've done, I'd truly appreciate your comments.Any ideas or additional questions are welcome.Cheers,Alberto

author: Berni
author_rank: Super Contributor
timestamp: August 29, 2023, 01:51:46 pm
content: I messed around with Cyclone V and i remember finding the HPS bus rather fiddly too.You really did need both ends of the bus started up properly before you do anything or it would just sit perpetually busy. Unfortunately i forgot the specific technical details since it was a while ago, but it is expected for it to be pretty picky and the documentation at the time was not all that clear of the exact right way to do it, so i had to do some trial and error to get it working.

author: betocool
author_rank: Regular Contributor
timestamp: August 29, 2023, 01:58:16 pm
content: Thanks,Another thing is that no matter how long the OS has been running, after a hard reboot it always fails. After a soft reboot, I can get the script started very quickly after all peripherals are up.Documentation for Altera these days is one of my pet hates. The Intel website requires a Microsoft login last I looked? And their forum is not terribly helpful at times.Cheers,Alberto

author: Berni
author_rank: Super Contributor
timestamp: August 29, 2023, 02:11:47 pm
content: I actually made a post on the Altera forums about it to help other people with getting this stubborn thing running. Looks like they imported it into the new forums:https://community.intel.com/t5/Nios-V-II-Embedded-Design-Suite/Guide-for-starting-HPS-FPGA-bridges-and-general-tips-in/m-p/163694/highlight/true

author: betocool
author_rank: Regular Contributor
timestamp: August 30, 2023, 01:55:19 pm
content: I saw that post, thanks for that! I haven't tried anything yet, but I started playing reading some registers.As you suggested there, the nrst_something ;-) register is "1" on the read and write for that hps-ram interface.I will have a closer look either tomorrow or Friday and see how things progress.Cheers,Alberto

author: tom66
author_rank: Super Contributor
timestamp: September 03, 2023, 12:10:26 pm
content: Where is the address for your data output set? An issue I've had in the past was that the Linux kernel had not reserved the block I had expected (misconfiguration in petalinux) which resulted in some kernel stuff ending up in the same area as the FPGA memory buffer. I was erasing a graphic framebuffer and thought it was truly bizarre that `ls -la` listed a load of files with 'UUUUUUUUUUU'... as the name until I realised my test pattern was 0x55 in the graphic buffer so the filesystem table in memory had been completely nerfed by my memory write.The kernel was surprisingly stable despite this (could start new processes by name even if listing cached directories failed), but trying to access any of the 'bad' file names, or 'cd' into any of the imaginary directories, resulted in an instahang.Anyway, I bring this up because if your HDL is expecting a read and write address to be set, could these be occasionally wrong, or could the kernel occasionally allocate into them?

author: betocool
author_rank: Regular Contributor
timestamp: September 04, 2023, 11:03:29 am
content: Thanks, the address is set to 0x30000000 and the VHDL code is reading it at address 0x6000000 from an 8-byte bus. 0x30000000 / 8 is 0x6000000. That seems to be ok.The issue here is that the error only happens at hard-reboot, not soft... mhhh... perhaps I should change the address to something higher where I'm sure the OS will not meddle? Worth a go. 0x30000000 is 50 MByte, I'm sure the other applications don't use that much at startup... but... I've been wrong in the past more often than I should!Other thing I've read in other posts is that the FPGA2SDRAM interface must be active, but I've checked all registers I can think of and they all look the same both after hard-boot and soft-boot.I've also followed all the FPGA compilation-> handoff file generation -> Uboot patching -> new image sequence and still the same.Yeah... tomorrow I will test the memory addressing, perhaps go up to over the 256Mbyte boundary.Cheers,Alberto

author: betocool
author_rank: Regular Contributor
timestamp: September 04, 2023, 11:06:03 am
content: Crap... had a missing 0! 0x3000000 is 50 odd MBytes, but 0x30000000 is 768 Mbytes... I'll still try though, why not?

author: tru
author_rank: Regular Contributor
timestamp: September 06, 2023, 10:40:14 am
content: I think it is limitation, of first time initialisation (cold boot) within linux of the FPGA2SDRAM interface, see old article here:https://www.intel.com/content/www/us/en/support/programmable/articles/000086918.htmlQuoteI think what you're seeing is that you are within linux and enabling the FPGA2SDRAM first time does enable it but will hang linux, due to SDRAM not being idle? Despite the hang, it is now in enabled state, a warm reset will keep the FPGA2SDRAM enabled, and so things work as expected after that.Perhaps you need to initialise it from U-Boot as a dummy first time, then within linux your script will re-initialise, but may be ok since it is not first time init.Note, if I remember correctly, U-Boot disables it just before loading linux kernel image. Also, in newer U-Boot the enable bridges command no longer enable the FPGA2SDRAM.

author: betocool
author_rank: Regular Contributor
timestamp: September 06, 2023, 12:32:19 pm
content: Thanks for that,I will have to re-enable access to u-boot to try that. Our project had the need for two UARTS and uboot had to go quiet. Still worth the effort.Changing the memory address didn't help either, but at least it didn't make matters worse.It really bugs me (boom-tish!) that Intel seems to hide this information in the dodgiest places and the newest information is from 2013. Ok, not true all of that, but yeah, I find working with Intel's way of doing things in the FPGA world terribly frustrating.I will continue having a look at this.Cheers,Alberto

author: betocool
author_rank: Regular Contributor
timestamp: September 11, 2023, 08:24:54 am
content: Thanks to all for the responses I got, eventually I got it to work.Like most of the comments before, the issue is as I understand it to re-start the FPGA2SDRAM interface after loading the FPGA rbf file. What worked and still works for me is adding following lines to the bootloader startup script:Code:enable_fpga2sdram=mw 0xFFC2505C 0xAbootcmd=run enable_fpga2sdram; run linux_load; bootz ${kernel_addr_r} - ${fdt_addr_r}It's not complete, but basically it's just writing the "applycfg" bit into the correct HPS register at address 0xFFC2505 (staticfg register).There is an excellent post about it here:https://github.com/zangman/de10-nano/blob/master/docs/FPGA-SDRAM-Communication_-Avalon-MM-Host-Master-Component-Part-3.mdSome of the other information is unfortunately outdated. The "enable_bridge_handoff" command (or so) doesn't exist anymore in the newer u-boot versions.Cheers,Alberto