Looking for help to debug 32F4 ethernet - Page 1
Forum section: electronics / general

author: peter-h
author_rank: Super Contributor
timestamp: July 25, 2024, 05:52:34 pm
content: I posted the background herehttps://www.eevblog.com/forum/programming/weird-strcat()-behaviour/Now I have narrowed it down to the ETH controller in the 32F417 not seeing any incoming data. This happens if a certain #included text string is defined as static const rather than just const. Yes this is totally bizzarre but I have to start somewhere.I can step through the ETH initialisation and that all seems to be OK. I haven't read it back to check every bit is correctly set - there are so many. The LAN8742 PHY also gets initialised via the "64 bit UART" interface.But is there some sort of low hanging fruit - a particular register which controls whether an incoming packet will get notified?For example this function is never seeing any dataCode:HAL_StatusTypeDef IF_HAL_ETH_GetReceivedFrame(ETH_HandleTypeDef *heth){uint32_t framelength = 0U;/* Check if segment is not owned by DMA *//* if (((heth->RxDesc->Status & ETH_DMARXDESC_OWN) == (uint32_t)RESET) && ((heth->RxDesc->Status & ETH_DMARXDESC_LS) != (uint32_t)RESET)) *///__DMB();if(((heth->RxDesc->Status & ETH_DMARXDESC_OWN) == (uint32_t)RESET)){/* Check if last segment */if(((heth->RxDesc->Status & ETH_DMARXDESC_LS) != (uint32_t)RESET)){/* increment segment count */(heth->RxFrameInfos).SegCount++;/* Check if last segment is first segment: one segment contains the frame */if ((heth->RxFrameInfos).SegCount == 1U){(heth->RxFrameInfos).FSRxDesc =heth->RxDesc;}heth->RxFrameInfos.LSRxDesc = heth->RxDesc;/* Get the Frame Length of the received packet: substruct 4 bytes of the CRC */framelength = (((heth->RxDesc)->Status & ETH_DMARXDESC_FL) >> ETH_DMARXDESC_FRAMELENGTHSHIFT) - 4U;heth->RxFrameInfos.length = framelength;/* Get the address of the buffer start address */heth->RxFrameInfos.buffer = ((heth->RxFrameInfos).FSRxDesc)->Buffer1Addr;/* point to next descriptor */heth->RxDesc = (ETH_DMADescTypeDef*) ((heth->RxDesc)->Buffer2NextDescAddr);/* Return function status */return HAL_OK;}/* Check if first segment */else if((heth->RxDesc->Status & ETH_DMARXDESC_FS) != (uint32_t)RESET){(heth->RxFrameInfos).FSRxDesc = heth->RxDesc;(heth->RxFrameInfos).LSRxDesc = NULL;(heth->RxFrameInfos).SegCount = 1U;/* Point to next descriptor */heth->RxDesc = (ETH_DMADescTypeDef*) (heth->RxDesc->Buffer2NextDescAddr);}/* Check if intermediate segment */else{(heth->RxFrameInfos).SegCount++;/* Point to next descriptor */heth->RxDesc = (ETH_DMADescTypeDef*) (heth->RxDesc->Buffer2NextDescAddr);}}/* Return function status */return HAL_ERROR;}This lineif(((heth->RxDesc->Status & ETH_DMARXDESC_OWN) == (uint32_t)RESET))never becomes True.I am suspecting that initialised constants are getting corrupted somewhere. Or maybe there is a buffer alignment issue with DMA but I have checked hopefully everything. It is 4-aligned.The rx is polled from an RTOS task; it is not interrupt-driven.

author: glenenglish
author_rank: Frequent Contributor
timestamp: July 25, 2024, 08:46:38 pm
content: Do you have link up and link down notifications , and negotiation information from the MDIO interface ?What happens if you try and send a packet ?

author: peter-h
author_rank: Super Contributor
timestamp: July 25, 2024, 09:31:38 pm
content: Yes I do have Link Up/Down detection.There is this macro for Link detect#define netif_is_link_up(netif) (((netif)->flags & 0x04U) ? (u8_t)1 : (u8_t)0)From stepping through I don't ever see Link Up. But doesn't LU need some incoming data? This isn't like USB where you can sense VBUS (for a simple check). You need a cable going to a switch sending out some packets. I have that connection.Obviously this is pretty basic. I am suspecting an issue with initialised data (whether const, or copied flash -> ram) so the init is not happening correctly.Putting that static const string at the end of another very similar file containing another static const string doesn't bomb it, suggesting that any movement in memory addresses is not the problem.

author: glenenglish
author_rank: Frequent Contributor
timestamp: July 26, 2024, 04:19:37 am
content: If you dont have link up, you wont get any packets.....

author: peter-h
author_rank: Super Contributor
timestamp: July 26, 2024, 07:05:21 am
content: Link Up must be the ETH subsystem receiving some packet and setting a register bit(netif)->flags & 0x04USince the cable etc is still there, this must be a bad initialisation of the subsystem (the 32F or the LAN8742).I will try to probe the LAN8742 "UART" interface.

author: glenenglish
author_rank: Frequent Contributor
timestamp: July 26, 2024, 10:32:40 am
content: Have a read up on how the PHY protocol works on the MDIO. read about negotiation, advertising etc.Link up is not the ETH subsytem receiving a packet.

author: peter-h
author_rank: Super Contributor
timestamp: July 26, 2024, 02:33:14 pm
content: Fortunately I am changing the direction of this because I found that (of ETH and USB both getting buggered by the use of a "const" on a string, GCC v11) it buggers up USB even if all the ETH code is stripped out.It is 99% sure that whatever the problem is, it is nothing to do with ETH (unless it is a buffer alignment issue or some such).Unfortunately USB is also mega complex. I am suspecting peripheral initialisation is going wrong, in particular some variable likeint fred = 5;getting messed up.

author: ajb
author_rank: Super Contributor
timestamp: July 26, 2024, 04:06:24 pm
content: Quote from: peter-h on July 26, 2024, 07:05:21 amThe link is established at the PHY level, not the MAC level (which is the part that's built into the MCU). If their pins are strapped correctly, two PHYs can negotiate a link with no MACs/MCUs involved at all. The host can determine if the link is up (and at what speed, etc) by querying the PHY's status register through the SMI, which is (usually) the MDIO/MDC interface (but can also be an SPI interface, especially if the PHY is actually a whole switch with many ports plus switching behavior to configure).Assuming the STM32F4 has the same MAC hardware as the STM32F7 series, one thing to note is that the MAC must be prompted to scan the descriptor list to see if any are available to receive into. If it scans the list and finds no descriptors it owns, it goes into idle mode and won't receive anything. If things get initialized out of order, it's possible for the MAC to scan the list, find no descriptors available, and then go idle before the descriptors are fully initialized. This also means your polling function (or the one provided by HAL) must prompt the MAC to rescan the descriptor list after the buffers are consumed, handled, and returned to the MAC's ownership.

author: ejeffrey
author_rank: Super Contributor
timestamp: July 27, 2024, 01:50:39 am
content: It sure sounds like a buffer overflow. Probably by one or maybe two bytes. Changing the size or storage location of specific variables can affect the number of padding bytes between subsequent fields. If you overflow into a padding byte it's probably harmless, if you overflow into another data structure, bad news.Write overflows are the most obvious but a read overflow can also cause problems since padding bytes are usually initialized to null.I would look in the map file and find the variable whose size change triggers the bug, then look at the nearby objects to see if one looks like a good culprit for an overflow.One thing you could do is take a "working" build and find the locations of every padding byte in the data section. Write random data into the padding bytes then run and see if any get overwritten, or if the build now crashes.

author: peter-h
author_rank: Super Contributor
timestamp: July 27, 2024, 08:12:10 am
content: The curious thing is that it affects USB and ETH, while other 20+ RTOS tasks continue running.I've just realised that the one thing these two have in common is the generation of special clocks. USB uses 48MHz and ETH uses 50MHz. But the 50MHz comes from the PHY chip...Both also use DMA and that does need aligned addresses (unless used in byte mode). I have a lot of stuff like thisstatic uint8_t cdc_receive_temp_buffer __attribute__ ((aligned (4)));and there is another syntaxALIGN_BEGIN static uint8_t USBD_MSC_CfgHSDesc __ALIGN_END =The latter uses a macroCode:/* In HS mode and when the DMA is used, all variables and data structures dealingwith the DMA during the transaction process should be 4-bytes aligned */#if defined (__GNUC__) /* GNU Compiler */#define __ALIGN_END __attribute__ ((aligned (4)))#define __ALIGN_BEGIN#else#define __ALIGN_END#if defined (__CC_ARM) /* ARM Compiler */#define __ALIGN_BEGIN __align(4)#elif defined (__ICCARM__) /* IAR Compiler */#define __ALIGN_BEGIN#elif defined (__TASKING__) /* TASKING Compiler */#define __ALIGN_BEGIN __align(4)#endif /* __CC_ARM */#endif /* __GNUC__ */I can change the failure by moving that codeCode:// Function returns a pointer to the customer application string.// This is the customer application name - max 32 bytes#include "appname.ini"char * get_appname(void){return (char*) (appnamestring);}around within the project, but ETH+USB do both fail together.In fact just changing the size of the name string also controls it (3 bytes breaks it, any more makes it run). So I have a very easy way to break it or notMore weird is that this has been in development for years (I have some 400 snapshots backed up, over just 2 years) and has never done anything "funny". One thing I looked at again was the linkfile...

author: peter-h
author_rank: Super Contributor
timestamp: July 27, 2024, 03:24:02 pm
content: I've fixed it, by stepping back through previous versions. Can anyone see a problem with this linkfile syntax? The purpose is to first collect DATA from the 1st module (main.o) and then collect DATA from the remaining modules. There is no particular reason for doing that other than it is neater in the .map fileCode:/* Initialized data sections for non boot block code. These go into RAM. LMA copy is loaded after code. *//* This stuff is copied from FLASH to RAM by C code in the main stub *//* This is stuff like int fred = 1; which is copied flash -> ram (not const) */. = ALIGN(4);/* main.c DATA */.main_data :{. = ALIGN(4);_s_nonboot_data = .;/* create a global symbol at data start */*main.o (.data .data*)/* .data sections */. = ALIGN(4);} >RAM AT >FLASH_APP/* Remaining DATA */._other_data :{. = ALIGN(4);*(.data .data*)/* .data sections */. = ALIGN(4);_e_nonboot_data = .;/* define a global symbol at data end */} >RAM AT >FLASH_APP/* used by the main stub C code to initialize data */_si_nonboot_data = LOADADDR(.main_data);Previously it was done all in one go and that works fineCode:.all_nonboot_data :{. = ALIGN(4);_s_nonboot_data = .;/* create a global symbol at data start */*(.data .data*)/* .data sections */. = ALIGN(4);_e_nonboot_data = .;/* define a global symbol at data end */} >RAM AT >FLASH_APP/* used by the main stub C code to initialize data */_si_nonboot_data = LOADADDR(.all_nonboot_data);Obviously it would be good to know what the problem is in the first one, but the linkfile syntax is horrible and there is no useful error reporting; you just get a line # because real men don't need anything elseI did always suspect the issue was with DATA not getting initialised correctly in RAM from its FLASH image.The code in startup.s which sets up the initialised data is the standard ST codeCode:/* Copy the boot block initialised data from FLASH to SRAM */movs r1, #0b LoopCopyDataInitCopyDataInit:ldr r3, =_si_boot_dataldr r3, [r3, r1]str r3, [r0, r1]adds r1, r1, #4LoopCopyDataInit:ldr r0, =_s_boot_dataldr r3, =_e_boot_dataadds r2, r0, r1cmp r2, r3bcc CopyDataInitElsewhere I did it in C, with memcpy() etc but in this case I left the old startup code.

author: Siwastaja
author_rank: Super Contributor
timestamp: July 27, 2024, 03:44:17 pm
content: Quote from: peter-h on July 25, 2024, 05:52:34 pmAs others have repeatedly said, you have not narrowed down anything. You are probably seeing effects of memory corruption, and any random change in code affects what exactly breaks at that time.The problem isn't where it breaks down.Therefore, usual strategy of narrowing down exact place of breakage and then trying to modify that place is wasted time, because the reason is totally elsewhere.But the place which breaks can still act as a hint.I suggest things like:* Look at static code analysis tools* List symbol addresses and look what iscloseto the weirdly behaving variable, for over/underindexing* do text search on memcpy and go through every single one of them, carefully checking the addresses and size parameters* same for array indexing []This is all tedious and slow work, but one thing is sure: wasting time in things thatdefinitelyare not the problem is 100% waste of time.

author: peter-h
author_rank: Super Contributor
timestamp: July 27, 2024, 03:54:27 pm
content: What do you think of that linkfile code?

author: Doctorandus_P
author_rank: Super Contributor
timestamp: July 29, 2024, 07:15:23 am
content: I agree with Siwastaja. Bugs that appear and disappear by shifting things around in memory (variable declaration, linkfile settings) are often a result of memory corruption. An uninitialized pointer writing to an already occupied position. Writing to one cell beyond an array size. That sort of thing. This means you can not find such bugs by examining the variables / memory that is being corrupted, but you have to figure out from where that memory is being overwritten.I once spent 8 work days narrowing down a bug like this. My job was not to find the bug itself, but to delete sections of code while still being able to reproduce the bug. In those 8 days I managed to delete around 90% to 95% of the code. After that I got stuck and a colleague with more experience took over. With help of a logic analyzer (with built in dissassembler) he was able to figure out a few days later that it was a pointer that was only capable to iterate over an 64k byte memory area being used in a bigger chunk of memory, and it was fixed by changing a compiler setting for a particular pointer size.

author: peter-h
author_rank: Super Contributor
timestamp: July 29, 2024, 07:48:43 am
content: QuoteYeah - doing the same here. Removed rather more than 95% and can still replicate it (on the old code - I decided to have a closer look). I am still convinced it is a linkfile issue.

author: peter-h
author_rank: Super Contributor
timestamp: July 29, 2024, 03:29:18 pm
content: Found it... those ALIGN 4 directives create gaps (obviously) and these create an out of sync situation between the FLASH based data and the RAM addresses to which the FLASH based data is getting copied to at startup.Teaches one to not be too clever and try to order modules sensibly! Although one can still achieve that quite simply as I posted herehttps://www.eevblog.com/forum/programming/a-specific-q-on-gcc-linker-script-syntax/Align directives are implemented correctly within modules but if you have multiple sections in a linkfile (for multiple modules) then sync is lost. This is rather weird but probably does not catch out many people because who would bother to do it that way.