Anyone here familiar with LWIP? - Page 1
Forum section: electronics / general

author: peter-h
author_rank: Super Contributor
timestamp: July 18, 2022, 01:20:41 pm
content: This is a weird one. I posted it on the ST forum but nobody there replies (a huge number of posts every day).To interface LWIP to the low level ETH hardware, one has a file normally called ethernetif.c. This contains low_level_input() and low_level_output() functions. In the simplest case (mine) these are polled, not interrupt-driven.My latter function - this code is all over the internet, in pretty well identical form - isCode:static err_t low_level_output(struct netif *netif, struct pbuf *p){err_t errval;struct pbuf *q;uint8_t *buffer = (uint8_t *)(EthHandle.TxDesc->Buffer1Addr);__IO ETH_DMADescTypeDef *DmaTxDesc;uint32_t framelength = 0;uint32_t bufferoffset = 0;uint32_t byteslefttocopy = 0;uint32_t payloadoffset = 0;DmaTxDesc = EthHandle.TxDesc;bufferoffset = 0;/* copy frame from pbufs to driver buffers */for(q = p; q != NULL; q = q->next){/* Is this buffer available? If not, goto error */if((DmaTxDesc->Status & ETH_DMATXDESC_OWN) != (uint32_t)RESET){errval = ERR_USE;goto error;}/* Get bytes in current lwIP buffer */byteslefttocopy = q->len;payloadoffset = 0;/* Check if the length of data to copy is bigger than Tx buffer size*/while( (byteslefttocopy + bufferoffset) > ETH_TX_BUF_SIZE ){//osDelay(2); // TODO/* Copy data to Tx buffer*/memcpy( (uint8_t*)((uint8_t*)buffer + bufferoffset), (uint8_t*)((uint8_t*)q->payload + payloadoffset), (ETH_TX_BUF_SIZE - bufferoffset) );/* Point to next descriptor */DmaTxDesc = (ETH_DMADescTypeDef *)(DmaTxDesc->Buffer2NextDescAddr);/* Check if the buffer is available */if((DmaTxDesc->Status & ETH_DMATXDESC_OWN) != (uint32_t)RESET){errval = ERR_USE;goto error;}buffer = (uint8_t *)(DmaTxDesc->Buffer1Addr);byteslefttocopy = byteslefttocopy - (ETH_TX_BUF_SIZE - bufferoffset);payloadoffset = payloadoffset + (ETH_TX_BUF_SIZE - bufferoffset);framelength = framelength + (ETH_TX_BUF_SIZE - bufferoffset);bufferoffset = 0;}/* Copy the remaining bytes */memcpy( (uint8_t*)((uint8_t*)buffer + bufferoffset), (uint8_t*)((uint8_t*)q->payload + payloadoffset), byteslefttocopy );bufferoffset = bufferoffset + byteslefttocopy;framelength = framelength + byteslefttocopy;}/* Prepare transmit descriptors to give to DMA */sys_mutex_lock(&lock_eth_if_out);HAL_ETH_TransmitFrame(&EthHandle, framelength);sys_mutex_unlock(&lock_eth_if_out);errval = ERR_OK;error:/* When Transmit Underflow flag is set, clear it and issue a Transmit Poll Demand to resume transmission */if ((EthHandle.Instance->DMASR & ETH_DMASR_TUS) != (uint32_t)RESET){/* Clear TUS ETHERNET DMA flag */EthHandle.Instance->DMASR = ETH_DMASR_TUS;/* Resume DMA transmission*/EthHandle.Instance->DMATPDR = 0;}return errval;}Note the commented-out osDelay(2). This fixed a bug whereby a large data transfer (2MB) was fairly often corrupted. An examination of the data found that the size was preserved but some data appeared in 2 places consecutively, as if a buffer was being written before previous data was extracted out of it (by the dedicated DMA controller which services the 32F4 ETH subsystem).I suspected the memcpy was running while the DMA was still reading that buffer. That delay immediately fixed the issue. I use (2) rather than (1) because osDelay(1) is actually 0ms to 1ms.The obvious Q is: how come this works for other people? Or maybe lots of people fixed this bug and never posted about it. OTOH I can easily see that most users would never discover it, because it shows up only on- big transfers (1MB+)- data source is very fast (my flash FS read speed is > 1mbyte/sec)If point #2 is not the case then this won't show up because LWIP will be feeding the packets to low_level_output too slowly and the DMA will always be under-running. Those people are wasting RAM having more than 1 TX buffer anywhere, too, so all the stuff all over the internet about "tuning TCP" is wastedMy first suspicion was that this bit of code (which AIUI is supposed to check if there is a DMA TX buffer available) is supposed to bebeforethat memcpy which is above it, but actually if one looks at the program flow, it should be ok because it is repeated elsewhere.Code:/* Check if the buffer is available */if((DmaTxDesc->Status & ETH_DMATXDESC_OWN) != (uint32_t)RESET){errval = ERR_USE;goto error;}My other thought was: why return an error code if no TX buffer available when a) the TX DMA cannot fail (unless the silicon is duff); b) at 10/100mbps, a buffer will become available very fast; c) the error condition is returned to LWIP and according to google (there is virtually zero support on LWIP anywhere, even on LWIP mailing lists) LWIP does not always retransmit on a TX error.Anyway, the osDelay(2) is a bad bodge, so I fixed it inside HAL_ETH_TransmitFrame() by waiting for the DMA status to show "all transfers complete", withCode:// Make this function blocking, otherwise following code overwrites the last DMA buffer!if ( (((heth->Instance)->DMASR) & (0x7 << 20)) != 0 ){taskYIELD(); // not really necessary since the time here would be a max of 1 MTU at 10-100mbps}which is probably suboptimal because it prevents LWIP getting the next packet ready while DMA is transmitting the previous one to ETH. With some test code, the output speed is about 200kbytes/sec which is totally fine for the application.I also found that replacing the "check if buffer is available" withCode:// Is this buffer available? If not, waitwhile ( (DmaTxDesc->Status & ETH_DMATXDESC_OWN) != 0 ) {}works too, and the function never returns an error code.Does anyone know anything about this stuff?

author: ttt
author_rank: Regular Contributor
timestamp: July 18, 2022, 08:34:27 pm
content: The STM32F4 series have I+D caches. In case you have that enabled have you tried to flush the cache before the memcpy (setting DCRST and ICRST bits in FLASH_CR)?

author: wek
author_rank: Frequent Contributor
timestamp: July 18, 2022, 10:32:49 pm
content: Quote from: ttt on July 18, 2022, 08:34:27 pmThose caches are on the FLASH interface. There's no reason to flush them unless you change the FLASH content (i.e. reprogram FLASH), and it's not the case here.JW

author: wek
author_rank: Frequent Contributor
timestamp: July 19, 2022, 12:13:39 am
content: QuoteDropped frames should result in data missing, not corrupted.And IP should be OK with missing data, that's why TCP.I don't offer answers, just doubts.JW

author: peter-h
author_rank: Super Contributor
timestamp: July 19, 2022, 09:52:14 am
content: QuoteI found that too but all I could find is that the H7 has this issue, not the F4 - asweksays above.QuoteI don't think frames are dropped. The packet serial numbering system would pick that up. I think, with my very limited knowledge of TCP/IP, that if you overwrite that buffer with "junk"and the buffer was holding a complete packet, then the ETH controller will generate a "good" CRC on the packet (this I believe is a low level hardware feature; LWIP is not computing a CRC32, is it?) as it is being transmitted. And since there is no end-to-end CRC, the corruption is not detected.Possibly my bold text above is irrelevant and actually each of the buffers does simply get sent as a packet on ETH, with a 1:1 buffer-packet relationship. I know the 32F4 ETH controller picks up buffers according to order in some sort of list, but I don't think it concatenates them to fill up an MTU.For amusement: Many years ago I designed a token-ring LAN (before ETH was cost-effective or even less than unbelievably complicated, using a Z180+85c30, SDLC packets, and a Manchester encoded isolated MIL1553 physical interface). I did that after a contracted-out LAN project with the WD2840 token ring controller ground to a halt (£500/day in 1985!) and I did it myself the hard way. I had issues like this too. With just CRC16, errorswouldslip through if there was enough noise.

author: wek
author_rank: Frequent Contributor
timestamp: July 19, 2022, 11:17:23 am
content: > ETH controller will generate a "good" CRC on the packet (this I believe is a low level hardware feature; LWIP is not computing a CRC32, is it?If you've set so.The initial post inhttps://community.st.com/s/question/0D53W00001bKph9SAC/hardware-ipv4-checksum-on-an-stm32f407-is-not-working-though-all-the-proper-settings-are-set-works-on-rare-occasions-oddlysums it up.> if you overwrite that buffer with "junk"The question is, where does the buffer overwrite happen, exactly. It might've happened at the low_level_output() routine - which would imply that the OWN bit checking does not work as intended, for whatever reason; but also before that, still in the IP/TCP stack.JW

author: peter-h
author_rank: Super Contributor
timestamp: July 19, 2022, 12:01:19 pm
content: Thank you.Looking in opt.h and lwipopts.h, the way this is supposed to work is that opt.h does the defines for lwip and lwipopts.h overrides those defined in lwipopts.h. Looking at both files, in my project, it looks like onlyipv6checksums are done in software. I am not supporting ipv6 anyway.QuoteI haven't got a clue how or where to zero that value. However, this sounds like a consistent thing, so it should be easy to reproduce.QuoteIndeed. I think it was the memcpy that was doing the overwrite, since delaying just before it fixed the problem. But it is very marginal; a board I am working on, with a load of RTOS tasks running, doesn't show it. I am transferring 2MB jpegs as the easiest way to see data corruption and haven't seen this problem. I will come back to it later.As an interesting aside, ref this#define ETH_TXBUFNB 2 // (4U) /* 4 Tx buffers of size ETH_TX_BUF_SIZE */I have found that with 2 or anything bigger I am getting a tx speed of120kbytes/sec, with 1 it falls to half that. The flash file read speed is about 5x to 10x bigger so there is clearly a big bottleneck in LWIP. For this application this is fine but maybe it is indicative of something? Removing the above mentioned wait for ETH DMA to be totally finished after each transfer (making it blocking) also makes no difference. Increasing MEM_SIZE in lwipopts.h dramatically (from 6k to say 16k) improves the tx speed only 20%. I will post this in the other thread.

author: peter-h
author_rank: Super Contributor
timestamp: July 21, 2022, 02:25:56 pm
content: This issue is very hard to reproduce. Now, it is gone. I have changed a few things around. It was seen only on a board running a version of the software with most RTOS tasks stripped out (which is what I might expect, because it looks like LWIP is feeding in the packets extra fast) and I tried that too, without success.So I left in that bit of code which checks the 000 "DMA all finished" bits. It makes no difference to performance that I can see.

author: betocool
author_rank: Regular Contributor
timestamp: July 21, 2022, 11:16:23 pm
content: I've had failures and wins with LWIP, on F7s and H7, and long long time ago on an Atmel UC100 or something along those lines, 2009.I don't have the references here unfortunately, but what I do to increase throughput is to increase the size of the receive buffer to, say, 1024 bytes or even 1536 (I think that is the max Ethernet transfer size), as well as increasing the number of buffers used for LwIP on the config files. In an F7 or H7 memory is not a problem usually, so I allocate a lot.I had one application where I used TCP/IP raw (bare metal), and it worked, but it took a lot of tinkering really to get it to the speeds I needed.After that, I tried using TCP/IP with FreeRTOS, and again after some tinkering I got it working steady. I never tested it more than a few hours at a time though, but speeds about 10 MBit or so were ok. The important thing here, at least for me, was to have a good enough data receive buffer before writing data from the Micro to the PC, for that I used a reasonably large StreamBuffer.Also, last thing, I always prefer to set up the micro as a server, and any application as clients on PCs. I feel the micro doesn't really know where he is sending the data to a priori. I usually use micros as data gatherers, eventually some processing, before bulk sending to a PC. I have not come across an application where a PC would have to send tons of data to a micro.My 2c's worth.Cheers,Alberto

author: peter-h
author_rank: Super Contributor
timestamp: July 22, 2022, 08:21:22 am
content: I have been tinkering with it and (in retrospect not surprisingly) found a huge bottleneck in the polled low level receiveCode:/*** This function is the ethernetif_input task. It uses the function low_level_input()* that should handle the actual reception of bytes from the network* interface. Then the type of the received packet is determined and* the appropriate input function is called.** This is a standalone RTOS task so is a forever loop.* Could be done with interrupts but then we have the risk of hanging the KDE with fast input.**/void ethernetif_input( void * argument ){struct pbuf *p;struct netif *netif = (struct netif *) argument;// This mutex position is based on an idea from Piranha here, with the delay added// https://community.st.com/s/question/0D50X0000BOtUflSQF/bug-stm32-lwip-ethernet-driver-rx-deadlockdo{sys_mutex_lock(&lock_eth_if_in);p = low_level_input( netif );sys_mutex_unlock(&lock_eth_if_in);if (p!=NULL){if (netif->input( p, netif) != ERR_OK ){pbuf_free(p);}}osDelay(10);// this has a dramatic effect on ETH speed, both ways} while(true);}This controls both up and down speed because TCP/IP is a duplex flow.I am getting 250kbytes/sec which is plenty for this application. By going for osDelay(1) I get 1200kbytes/sec and one could probably get a bit more by making RX interrupt-driven but then you need to be careful with mutexes (can't call them from an ISR). So that's 10mbps too, like you got. But my LAN connection is 100mbps and I think the 32F4 ETH controller processes the packets so fast that having lots of low level buffering is wasted. In fact I get the same speed with just 1 low level (ETH DMA) buffer; now I am using 2 for TX and 2 for RX.The above code is transferring 2500 bytes every 10ms. My experiments suggest that tweaking the lwipopts.h buffer numbers etc makes almost no difference at these low packet rates.There are also some very basic points which you don't read about much. For example, the ETH running at 100mbps (most gear is 1000mbps but your 32F4 ETH is 10 or 100 only, so it will run at 100 unless you deliberately cripple it) and yourTXpackets will be disappearing down the wire so fast that it makes almost no difference how much or how little buffering you have set up. No point in having 10 packet buffers (15k of RAM) if only 1 ever gets used; one discovers this only by prefilling the buffer area with say 0xAA and then after a while taking a look at it. I reckon in a typical RTOS based scenario where you have a load of other code running, anything more than 2TXbuffers in ETH and 2 in LWIP will be wasted - in most applications.

author: arijav
author_rank: Contributor
timestamp: July 22, 2022, 07:26:39 pm
content: I am sorry to be slightly out of topic, but I would like to give you a recommendation. After months trying to get a reliable ETH + Raw API LWIP (w/o OS) setup with an STM32H7 for an application that required very high availability (sending high speed tcp data streams over weeks in some cases months) I gave up, as after 10 hours aprox. the STM always stopped sending packets. I have moved now to a different stack, FreeRTOS+TCP, and it is a real breeze, the connection is fully reliable.https://www.freertos.org/FreeRTOS-Plus/FreeRTOS_Plus_TCP/index.htmlIf you need high availability I really recommend to move away from the buggy ETH+LWIP STM32 setup.

author: artag
author_rank: Super Contributor
timestamp: July 22, 2022, 07:52:28 pm
content: I don't know that this is relevant, and it refers to an LWIP implementation I did on a PPC555 some tears ago. But it did have a cause that surprises me.I would get errors (exceptions) on transmit frames. The cause of these was DMA underrun, and it occurred when the memory bus was busy with some other DMA transfers. It served as a warning that DMA is not 'free performance' just because the CPU isn't doing it. Even though the DMA bus was wide, relatively high performance, and running between an internal peripheral and internal (afaicr) RAM buffers, it was possible even with 10Mb ethernet to use up all the bandwidth and both the concurrent transfers couldn't keep to their timing.

author: peter-h
author_rank: Super Contributor
timestamp: July 22, 2022, 09:20:03 pm
content: I think so much depends on some small details.I have FreeRTOS running a load of tasks, some quite complicated e.g. DMA->DACs waveform generator, and I have five ETH-using tasks. One is doing ICMP pings, one is an HTTP server, and two are doing complicated HTTPS/TLS data uploads to servers. The 5th is NTP (UDP packets).It runs solidly. However, I do get crashes when the HTTP server is running. After a number of hours only. Here:https://www.eevblog.com/forum/microcontrollers/32f4-hard-fault-trap-how-to-track-this-down/msg4315627/#msg4315627The HTTP server is the only one using the LWIP "netconn" API. The others are using the RAW or Socket APIs. I have set up LWIP to be thread-safe (lwipopts.h) and that seems to work.So, yeah, there may be bugs in there, but there can be bugs anywhere, and in today's fast moving world, how good software is depends primarily on a) how many people use it b) how varies are their applications c) whether the developer is responsive. I reckon most open source software is actually pretty crap and requires weeks or months of trawling the internet and patching bugs.Artag - how did you solve that one?

author: artag
author_rank: Super Contributor
timestamp: July 22, 2022, 10:27:48 pm
content: Quote from: peter-h on July 22, 2022, 09:20:03 pmI think we had to protect the ethernet transfer, which had the most stringent timing requirements. But I forget what the conflict was with - it may have been with the cache fill. Though that doesn't really add up with the use of internal memory. Maybe it was an external bandwidth conflict and we buffered the ethernet packet through internal ram to give it more bandwidth.It was 15 years ago and I haven't worked there for much of that time !

author: peter-h
author_rank: Super Contributor
timestamp: July 23, 2022, 08:07:56 am
content: LWIP, like TLS, FreeRTOS, and everything supplied by STM with Cube IDE, must have had hundreds of bug fixes just in the last few years. By the time I reach my actuarial life expectancy, 95.3% of the bugs will have been fixed, and the remaining 4.7% will remain because the coders have got themselves girlfriends

author: peter-h
author_rank: Super Contributor
timestamp: January 14, 2023, 04:29:11 pm
content: Back to this, and looking for ideas...I have a couple of my boxes, set up for serial - ETH, back to back, looping back all the way to the start. Sending test packets with a special tester.It all works. Transit time is 10-20ms which is fully accounted for with my code, but I am finding that very occassionally the delay goes to 1-2 seconds.I am running on a LAN so the ETH section is just a switch.Does anyone have any idea on what a TCP/IP stack (LWIP in this case) could be causing these long gaps?I don't know of any way to troubleshoot LWIP. It is open source but the devs abandoned it > 10 years ago, so a google finds a huge number of hits and few solution. The usual stuff

author: 8goran8
author_rank: Contributor
timestamp: January 14, 2023, 08:13:59 pm
content: I would try to sniff the ethernet traffic with wireshark and observe the IP packets just before the delay. Perhaps some comparison could be made with recorded content when there is no lag. That might give me an idea where to start looking for the problem.

author: gmb42
author_rank: Frequent Contributor
timestamp: January 16, 2023, 11:34:40 am
content: Quote from: 8goran8 on January 14, 2023, 08:13:59 pmIf you're at all concerned about accurate timing of packets you should consider using aTAP. If the price of a TAP is too rich for your tastes then you could use a span or mirror port on a switch that supports such options but you will then lose some accuracy in timing.An on-machine capture is the least reliable option for accurate timing, especially as off-loading and other NIC acceleration features can create a false picture of what actually goes out on the wire.

author: dut:Mark
author_rank: Newbie
timestamp: January 21, 2023, 12:14:24 pm
content: Quote from: peter-h on January 14, 2023, 04:29:11 pmJust FYI: latest 2.1.3 release dates from November 2021 as far as I can see.With respect to your issues and assuming you are using TCP (not UDP or raw Ethernet): think about what happens if you encounter packet drop, delayed ACKs, exhausted memory pools etc. Any broadcast traffic happening on your network which may affect resources consumed by your network stack? Is anything executing at your nodes that may block your network stack? Keep in mind that what happens on a single node, may affect the other nodes as well. And if you are only sending small packets, look into Nagle's algorithm and the silly window syndrome. As others mentioned, use Wireshark to investigate and keep in mind that TCP does not make any promises about time of delivery.

author: peter-h
author_rank: Super Contributor
timestamp: January 22, 2023, 11:11:07 am
content: I have disabled Nagle algorithm. Or at least I think I have - this is a part of the codeCode://set the socket to non-blockingint flags = O_NONBLOCK;ioctl(fd, FIONBIO, &flags);// Disable Nagle algorithm for faster responseflags = TCP_NODELAY;ioctl(fd, FIONBIO, &flags);It would have accounted for a lot of issues I was seeing. But ideally I would like to disable it globally for LWIP; the only code examples I can find disable it on a per-socket basis. Also, tracing through the LWIP code, I don't think the above is doing anything. After a lot of googling I thinkCode:// Disable Nagle algorithm for faster responseint optval = 1;int optlen = sizeof(optval);setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &optval, optlen);may be right. Stepping through it, it executes the right codeCode:#if LWIP_TCP/* Level: IPPROTO_TCP */case IPPROTO_TCP:/* Special case: all IPPROTO_TCP option take an int */LWIP_SOCKOPT_CHECK_OPTLEN_CONN_PCB_TYPE(sock, optlen, int, NETCONN_TCP);if (sock->conn->pcb.tcp->state == LISTEN) {return EINVAL;}switch (optname) {case TCP_NODELAY:if (*(const int*)optval) {tcp_nagle_disable(sock->conn->pcb.tcp);} else {tcp_nagle_enable(sock->conn->pcb.tcp);}LWIP_DEBUGF(SOCKETS_DEBUG, ("lwip_setsockopt(%d, IPPROTO_TCP, TCP_NODELAY) -> %s\n",s, (*(const int *)optval)?"on":"off") );break;The function which gets eventually called is lwip_setsockopt_impl() and when the correct parms are supplied this does the above tcp_nagle_disable() which does a#define tcp_nagle_disable(pcb) ((pcb)->flags |= TF_NODELAY)I wonder if anyone knows about this?The Nagle algorithm seems a throwback to old days of slow connections. It reportedly introduces delays of up to 300ms. Disabling it modifies packet timing dramatically.QuoteFor sure, but anyone who has the deep expertise to be able to do that and get useful information doesn't need to be asking questions on forumsThis stuff is probably the most complex stuff anyone is likely to be doing in embedded systems, and largely because one is trying to make use of an incredibly complicated and unsupported chunk of code called LWIP.Has anyone used the FreeRTOS+TCP stack mentioned above?https://www.freertos.org/FreeRTOS-Plus/FreeRTOS_Plus_TCP/index.html

author: peter-h
author_rank: Super Contributor
timestamp: January 24, 2023, 08:37:28 am
content: Not really expecting a reply because this stuff is specialised but having spent many days on one issue I think LWIP has a call rate (call frequency) limit on the socket calls. Probably writing but possibly also reading. If this is exceeded, it doesn't actually lose the data but some internal mechanism is triggered which inserts long delays of 1-2 secs in the data flow.Does anyone recognise this behaviour?

author: dare
author_rank: Contributor
timestamp: January 24, 2023, 03:51:23 pm
content: Quote from: peter-h on January 24, 2023, 08:37:28 amThis sounds like packet loss, but its hard to tell from the limited information available. It could be external, due to a lossy physical network, or it could be internal, due to buffer exhaustion.With its myriad of configuration options, LwIP can be rather inscrutable, but it does work. Your best bet is to enable LwIP internal logging and then run a packet capture of the conversation. I suggest configuring debug in lwipopts.h roughly as follows:Code:#ifdef LWIP_DEBUG#define MEMP_OVERFLOW_CHECK ( 1 )#define MEMP_SANITY_CHECK ( 1 )#define MEM_DEBUG LWIP_DBG_OFF#define MEMP_DEBUG LWIP_DBG_OFF#define PBUF_DEBUG LWIP_DBG_ON#define API_LIB_DEBUG LWIP_DBG_ON#define API_MSG_DEBUG LWIP_DBG_ON#define TCPIP_DEBUG LWIP_DBG_ON#define NETIF_DEBUG LWIP_DBG_ON#define SOCKETS_DEBUG LWIP_DBG_ON#define DEMO_DEBUG LWIP_DBG_ON#define IP_DEBUG LWIP_DBG_ON#define IP6_DEBUG LWIP_DBG_ON#define IP_REASS_DEBUG LWIP_DBG_ON#define RAW_DEBUG LWIP_DBG_ON#define ICMP_DEBUG LWIP_DBG_ON#define UDP_DEBUG LWIP_DBG_ON#define TCP_DEBUG LWIP_DBG_ON#define TCP_INPUT_DEBUG LWIP_DBG_ON#define TCP_OUTPUT_DEBUG LWIP_DBG_ON#define TCP_RTO_DEBUG LWIP_DBG_ON#define TCP_CWND_DEBUG LWIP_DBG_ON#define TCP_WND_DEBUG LWIP_DBG_ON#define TCP_FR_DEBUG LWIP_DBG_ON#define TCP_QLEN_DEBUG LWIP_DBG_ON#define TCP_RST_DEBUG LWIP_DBG_ON#define PPP_DEBUG LWIP_DBG_OFF#define LWIP_DBG_TYPES_ON (LWIP_DBG_ON|LWIP_DBG_TRACE|LWIP_DBG_STATE|LWIP_DBG_FRESH|LWIP_DBG_HALT)#endifIf you are unable to make sense of the packet capture and log output, share it here for others to interpret.

author: ejeffrey
author_rank: Super Contributor
timestamp: January 24, 2023, 04:34:31 pm
content: The Nagle algorithm is a congestion reduction algorithm to reduce the number of small packets, especially when doing things like sending unbuffered terminal or serial port data over a socket.The way it generally works is this: if you make a send() call that results in a short packet being sent, future send() calls will be buffered and not immediately sent until either 1) the previous packet is ACKed 2) you have accumulated enough data for a full size packet or 3) I think if the sender times out waiting for the ACK it will retransmit the old data as well as any new data.The Nagle algorithm should only really be an issue on high latency links, and can be avoided by batching your send calls manually. It also doesn't cause an issue in protocols that wait for a response from the other end to continue as they are limited by the round trip time anyway. You can turn it off with setsockopt but if your client doesn't try to send multiple short packets in a row that's unlikely to be the cause of your latency.It was historically a major problem for HTTP as the server, and to some extent the client would send headers first followed by the body and did not want to wait for a response in between.Also keep in mind that Nagle is a transmit side feature. You can enable/disable it on both sides independently, but one side can't force the other to change it's behavior.

author: ejeffrey
author_rank: Super Contributor
timestamp: January 24, 2023, 10:24:40 pm
content: You need to be looking at packet capture data from tcpdump/wireshark. There is basically no point in diagnosing any networking issue if you aren't 100% comfortable sorting through network capture data. Don't guess about broadcast traffic, traffic stalls, or other weirdness, measure it.If one or both endpoints are regular computers you can just run it on the endpoints. If both are embedded systems you will need to find a way to tap the signal. If your network hosts are 100 megabit, you can find an old "known hub", if you are operating at 1 Gb or higher you can use a switch with a MIRROR port or a dedicated tap device like this:https://www.amazon.com/Dualcomm-1000Base-T-Gigabit-Ethernet-Network/dp/B004EWVFAY/ref=asc_df_B004EWVFAY/Even just plugging a monitoring host into the same network switch as your devices would tell you about broadcast traffic.

author: peter-h
author_rank: Super Contributor
timestamp: January 24, 2023, 11:02:23 pm
content: My boxes are 10/100 and I do have a hub.Is there anything in LWIP debug output indicating packet re-transmission due to errors?What would be a broadcast used for, in the context of an embedded box? What data does LWIP return when it receives a broadcast packet?

author: dare
author_rank: Contributor
timestamp: January 25, 2023, 03:01:02 am
content: QoS support in the switch should make no difference. Its doubtful anything on your network is using QoS.You log output is jumbled and/or missing lines. For example, when an IP packet arrives, your log shows:Code:99586: ip4_input: packet accepted on interface st99587: | 4 | 5 | 0x00 | 40 | (v, hl, tos, len)99587: | 9821 |000| 0 | (id, flags, offset)99587: +-------------------------------+99587: | 192 | 168 | 3 | 64 | (dest)99587: pbuf_header: old 200050FE new 20005112 (-20)99587: | 10007 | 49154 | (src port, dest port)99587: | 0000029846 | (ack no)99587: | 5 | |010000| 1917 | (hdrlen, flags (99587: | 0xc2f3 | 0 | (chksum, urgp)99588: +-+-+-+-+-+-+-+-+-+-+-+-+-+- tcp_input: flags99588: State: ESTABLISHEDInstead, when a packet arrives, the LwIP log output should look like this:Code:tcpip_thread: PACKET 0x563fa1ea8860ip_input: iphdr->dest 0xc801a8c0 netif->ip_addr 0xc801a8c0 (0x1a8c0, 0x1a8c0, 0xc8000000)ip4_input: packet accepted on interface tpip4_input:IP header:+-------------------------------+| 4 | 5 | 0x00 | 60 | (v, hl, tos, len)+-------------------------------+| 58354 |010| 0 | (id, flags, offset)+-------------------------------+| 64 | 6 | 0xd2af | (ttl, proto, chksum)+-------------------------------+| 192 | 168 | 1 | 1 | (src)+-------------------------------+| 192 | 168 | 1 | 200 | (dest)+-------------------------------+ip4_input: p->len 60 p->tot_len 60TCP header:+-------------------------------+| 44738 | 80 | (src port, dest port)+-------------------------------+| 0151356050 | (seq no)+-------------------------------+| 0000000000 | (ack no)+-------------------------------+| 10 | |000010| 64240 | (hdrlen, flags (SYN), win)+-------------------------------+| 0x2fcd | 0 | (chksum, urgp)+-------------------------------+tcp_input: packed for LISTENing connection.TCP connection request 44738 -> 80.Is your logging mechanism reliable? In particular, does it queue/block when log output is printed quickly? If not, you will need to fix that. Without reliable logging you will not be able to diagnose problems in LwIP.As others have said, full fidelity packet captures are also a must when diagnosing subtle networking issues. If you have a linux computer with two ethernet interfaces (e.g. a built-in ethernet port and a USB ethernet dongle) you can configure the computer to act as a bridge between the two network segments (with your embedded devices on either side of the bridge). Once this is configured, any traffic passing across the bridge be capture by a Wireshark instance running on the computer.All that said, the problem can likely be diagnosed via the LwIP logs, so get that working first.QuoteThere are various uses for broadcast/multicast in typical LANs. The most common examples are ARP and DHCP for IPv4, and Neighbor Discovery for IPv6. If your device speaks IPv4 only then it will need to support ARP and possibly DHCP. All other inbound broadcast/multicast packets will be received but should be ignored by LwIP.

author: ejeffrey
author_rank: Super Contributor
timestamp: January 25, 2023, 03:31:46 am
content: It sounds like you are only transmitting serial port speed over the network. Ordinary broadcast traffic shouldn't pose any problem at all.You say your device are 10/100.... Are any of your devices set to forced full duplex mode? If so, disable that. Set to auto negotiation or forced half duplex.

author: peter-h
author_rank: Super Contributor
timestamp: January 25, 2023, 08:58:00 am
content: The debugs come out via USB VCP which is very fast - hundreds of k bytes per second. But each line is truncated to 128 bytes. I can change that... I am logging the data with Teraterm, and flow control was disabled to make the debug function non-blocking if there was no terminal on the VCP. I have now enabled flow control and see that the traffic runs a bit slower, as expected.The ETH port is auto select, or should be. DHCP is currently off; I am using fixed IP. The two IPs are 192.168.3.64 and .68.Samples:Random place:Code:36805: State: ESTABLISHED36781: pbuf_alloc(length=182)36807: pbuf_alloc: allocated pbuf 2000575036807: pbuf_alloc(length=182) == 2000575036806: tcp_fasttmr: delayed ACK36809: pbuf_alloc(length=20)36810: pbuf_alloc(length=20) == 20002BD436810: tcp_output: sending ACK for 84298936811: pbuf_header: old 20002C08 new 20002BF4 (20)36811: ip4_output_if: st036812: IP header:36813: +-------------------------------+36813: | 4 | 5 | 0x00 | 40 | (v, hl, tos, len)36814: +-------------------------------+36815: | 17 |000| 0 | (id, flags, offset)36815: +-------------------------------+36816: | 255 | 6 | 0x0000 | (ttl, proto, chksum)36816: +-------------------------------+36817: | 192 | 168 | 3 | 64 | (src)36817: +-------------------------------+36818: | 192 | 168 | 3 | 68 | (dest)36818: +-------------------------------+36819: ip4_output_if: call netif->output()36820: pbuf_header: old 20002BF4 new 20002BE6 (14)36820: pbuf_free(20002BD4)36821: pbuf_free: deallocating 20002BD436821: tcpip_thread: PACKET 2000694036822: pbuf_header: old 20005760 new 2000576E (-14)36822: ip_input: iphdr->dest 0x4003a8c0 netif->ip_addr 0x4003a8c0 (0x3a8c0, 0x3a8c0, 0x40000000)36823: ip4_input: packet accepted on interface st36824: ip4_input:36825: IP header:36825: +-------------------------------+36826: | 4 | 5 | 0x00 | 168 | (v, hl, tos, len)36826: +-------------------------------+36827: | 1468 |000| 0 | (id, flags, offset)36827: +-------------------------------+36828: | 255 | 6 | 0x2dbf | (ttl, proto, chksum)36828: +-------------------------------+36829: | 192 | 168 | 3 | 68 | (src)36829: +-------------------------------+36830: | 192 | 168 | 3 | 64 | (dest)36830: +-------------------------------+36831: ip4_input: p->len 168 p->tot_len 16836831: pbuf_header: old 2000576E new 20005782 (-20)36832: TCP header:36833: +-------------------------------+36833: | 10007 | 49154 | (src port, dest port)36834: +-------------------------------+36835: | 0000842989 | (seq no)36835: +-------------------------------+36836: | 0000007406 | (ack no)36836: +-------------------------------+36837: | 5 | |011000| 2024 | (hdrlen, flags (36837: PSH36838: ACK36838:36839: ), win)36839: +-------------------------------+36840: | 0x3796 | 0 | (chksum, urgp)36841: +-------------------------------+36842: pbuf_header: old 20005782 new 20005796 (-20)36842: +-+-+-+-+-+-+-+-+-+-+-+-+-+- tcp_input: flags36843: PSH36844: ACK36844:36845: -+-+-+-+-+-+-+-+-+-+-+-+-+-+36845: State: ESTABLISHED36846: tcp_receive: window update 202436846: tcp_receive: pcb->rttest 0 rtseq 7150 ackno 740636847: tcp_output: nothing to send (00000000)36847: tcp_output: snd_wnd 2024, cwnd 7479, wnd 2024, seg == NULL, ack 740636848: State: ESTABLISHEDOne area hopefully captured during a traffic interruption:Code:109346: lwip_recvfrom: top while sock->lastdata=00000000109347: lwip_recvfrom(2): returning EWOULDBLOCK109350: lwip_recvfrom(0, 100053BC, 100, 0x8, ..)109350: lwip_recvfrom: top while sock->lastdata=00000000109351: lwip_recvfrom(0): returning EWOULDBLOCK109358: lwip_recvfrom(1, 20010088, 128, 0x0, ..)109358: lwip_recvfrom: top while sock->lastdata=00000000109359: pbuf_alloc(length=182)109360: pbuf_alloc: allocated pbuf 20004A70109360: pbuf_alloc(length=182) == 20004A70109361: tcpip_thread: PACKET 200068C8109361: pbuf_header: old 20004A80 new 20004A8E (-14)109362: ip_input: iphdr->dest 0x4003a8c0 netif->ip_addr 0x4003a8c0 (0x3a8c0, 0x3a8c0, 0x40000000)109362: ip4_input: packet accepted on interface st109363: ip4_input:109364: IP header:109364: +-------------------------------+109365: | 4 | 5 | 0x00 | 168 | (v, hl, tos, len)109365: +-------------------------------+109366: | 1968 |000| 0 | (id, flags, offset)109366: +-------------------------------+109367: | 255 | 6 | 0x2bcb | (ttl, proto, chksum)109367: +-------------------------------+109368: | 192 | 168 | 3 | 68 | (src)109368: +-------------------------------+109369: | 192 | 168 | 3 | 64 | (dest)109369: +-------------------------------+109370: ip4_input: p->len 168 p->tot_len 168109370: pbuf_header: old 20004A8E new 20004AA2 (-20)109371: TCP header:109372: +-------------------------------+109372: | 10007 | 49154 | (src port, dest port)109373: +-------------------------------+109374: | 0000888733 | (seq no)109374: +-------------------------------+109375: | 0000053510 | (ack no)109375: +-------------------------------+109376: | 5 | |011000| 2432 | (hdrlen, flags (109376: PSH109377: ACK109377:109378: ), win)109378: +-------------------------------+109379: | 0x86ed | 0 | (chksum, urgp)109379: +-------------------------------+109380: pbuf_header: old 20004AA2 new 20004AB6 (-20)109380: +-+-+-+-+-+-+-+-+-+-+-+-+-+- tcp_input: flags109381: PSH109382: ACK109382:109383: -+-+-+-+-+-+-+-+-+-+-+-+-+-+109383: State: ESTABLISHED109384: tcp_receive: pcb->rttest 0 rtseq 53406 ackno 53510109384: tcp_output: nothing to send (00000000)109385: tcp_output: snd_wnd 2432, cwnd 38837, wnd 2432, seg == NULL, ack 53510109385: State: ESTABLISHED109363: pbuf_alloc(length=182)109387: pbuf_alloc: allocated pbuf 200050E0109387: pbuf_alloc(length=182) == 200050E0109386: tcp_fasttmr: delayed ACK109389: pbuf_alloc(length=20)109389: pbuf_alloc(length=20) == 20002BD4109390: tcp_output: sending ACK for 888861109390: pbuf_header: old 20002C08 new 20002BF4 (20)109391: ip4_output_if: st0109392: IP header:109392: +-------------------------------+109393: | 4 | 5 | 0x00 | 40 | (v, hl, tos, len)109393: +-------------------------------+109394: | 610 |000| 0 | (id, flags, offset)109394: +-------------------------------+109395: | 255 | 6 | 0x0000 | (ttl, proto, chksum)109395: +-------------------------------+109396: | 192 | 168 | 3 | 64 | (src)109396: +-------------------------------+109397: | 192 | 168 | 3 | 68 | (dest)109397: +-------------------------------+109398: ip4_output_if: call netif->output()109398: pbuf_header: old 20002BF4 new 20002BE6 (14)109399: pbuf_free(20002BD4)109400: pbuf_free: deallocating 20002BD4109400: tcpip_thread: PACKET 20006940109401: pbuf_header: old 200050F0 new 200050FE (-14)109401: ip_input: iphdr->dest 0x4003a8c0 netif->ip_addr 0x4003a8c0 (0x3a8c0, 0x3a8c0, 0x40000000)109402: ip4_input: packet accepted on interface st109403: ip4_input:109404: IP header:109404: +-------------------------------+109405: | 4 | 5 | 0x00 | 168 | (v, hl, tos, len)109405: +-------------------------------+109406: | 1969 |000| 0 | (id, flags, offset)109406: +-------------------------------+109407: | 255 | 6 | 0x2bca | (ttl, proto, chksum)109407: +-------------------------------+109408: | 192 | 168 | 3 | 68 | (src)109408: +-------------------------------+109409: | 192 | 168 | 3 | 64 | (dest)109409: +-------------------------------+109410: ip4_input: p->len 168 p->tot_len 168109410: pbuf_header: old 200050FE new 20005112 (-20)109411: TCP header:109412: +-------------------------------+109412: | 10007 | 49154 | (src port, dest port)109413: +-------------------------------+109414: | 0000888861 | (seq no)109415: +-------------------------------+109416: | 0000053510 | (ack no)109416: +-------------------------------+109417: | 5 | |011000| 2432 | (hdrlen, flags (109417: PSH109418: ACK109418:109419: ), win)109419: +-------------------------------+109420: | 0x0171 | 0 | (chksum, urgp)109420: +-------------------------------+109421: pbuf_header: old 20005112 new 20005126 (-20)109421: +-+-+-+-+-+-+-+-+-+-+-+-+-+- tcp_input: flags109422: PSH109423: ACK109423:109424: -+-+-+-+-+-+-+-+-+-+-+-+-+-+109424: State: ESTABLISHED109425: tcp_receive: pcb->rttest 0 rtseq 53406 ackno 53510109425: tcp_output: nothing to send (00000000)109426: tcp_output: snd_wnd 2432, cwnd 38837, wnd 2432, seg == NULL, ack 53510109426: State: ESTABLISHED109372: lwip_recvfrom(0, 100053BC, 100, 0x8, ..)109428: lwip_recvfrom: top while sock->lastdata=00000000109429: lwip_recvfrom(0): returning EWOULDBLOCK109359: lwip_recvfrom(1): returning EWOULDBLOCK109431: lwip_recvfrom(2, 20010088, 128, 0x0, ..)109431: lwip_recvfrom: top while sock->lastdata=00000000109432: tcp_recved: received 128 bytes, wnd 2792 (128).109432: netconn_recv_data: received 20004A70, len=128109433: lwip_recvfrom: netconn_recv err=0, netbuf=20004A70109433: lwip_recvfrom: buflen=128 len=128 off=0 sock->lastoffset=0109434: lwip_recvfrom(2): addr=109435: 192.168.3.68109435: port=10007 len=128109436: lwip_recvfrom: deleting netbuf=20004A70109436: pbuf_free(20004A70)109437: pbuf_free: deallocating 20004A70109447: lwip_recvfrom(1, 20010088, 128, 0x0, ..)109447: lwip_recvfrom: top while sock->lastdata=00000000109447: lwip_recvfrom(1): returning EWOULDBLOCK109448: lwip_recvfrom(2, 20010088, 128, 0x0, ..)109448: lwip_recvfrom: top while sock->lastdata=00000000109449: tcp_recved: received 128 bytes, wnd 2920 (0).109450: lwip_recvfrom(0, 100053BC, 100, 0x8, ..)109451: lwip_recvfrom: top while sock->lastdata=00000000109452: lwip_recvfrom(0): returning EWOULDBLOCK109450: netconn_recv_data: received 200050E0, len=128109454: lwip_recvfrom: netconn_recv err=0, netbuf=200050E0109454: lwip_recvfrom: buflen=128 len=128 off=0 sock->lastoffset=0109455: lwip_recvfrom(2): addr=109456: 192.168.3.68109456: port=10007 len=128109457: lwip_recvfrom: deleting netbuf=200050E0109457: pbuf_free(200050E0)109458: pbuf_free: deallocating 200050E0109468: lwip_recvfrom(1, 20010088, 128, 0x0, ..)109468: lwip_recvfrom: top while sock->lastdata=00000000109468: lwip_recvfrom(1): returning EWOULDBLOCK109469: lwip_recvfrom(2, 20010088, 128, 0x0, ..)109469: lwip_recvfrom: top while sock->lastdata=00000000109470: lwip_recvfrom(2): returning EWOULDBLOCK109473: lwip_recvfrom(0, 100053BC, 100, 0x8, ..)109473: lwip_recvfrom: top while sock->lastdata=00000000109474: lwip_recvfrom(0): returning EWOULDBLOCKThis bit was captured during a period of a few seconds where traffic was being blocked by somethingCode:150169: lwip_recvfrom(1): returning EWOULDBLOCK150170: lwip_recvfrom(2, 20010088, 128, 0x0, ..)150170: lwip_recvfrom: top while sock->lastdata=00000000150171: pbuf_alloc(length=182)150172: pbuf_alloc: allocated pbuf 20004A70150172: pbuf_alloc(length=182) == 20004A70150171: tcp_recved: received 128 bytes, wnd 2920 (0).150173: tcpip_thread: PACKET 20006904150174: pbuf_header: old 20004A80 new 20004A8E (-14)150174: ip_input: iphdr->dest 0x4003a8c0 netif->ip_addr 0x4003a8c0 (0x3a8c0, 0x3a8c0, 0x40000000)150175: ip4_input: packet accepted on interface st150176: ip4_input:150177: IP header:150178: +-------------------------------+150178: | 4 | 5 | 0x00 | 168 | (v, hl, tos, len)150179: +-------------------------------+150180: | 2206 |000| 0 | (id, flags, offset)150180: +-------------------------------+150181: | 255 | 6 | 0x2add | (ttl, proto, chksum)150181: +-------------------------------+150182: | 192 | 168 | 3 | 68 | (src)150182: +-------------------------------+150183: | 192 | 168 | 3 | 64 | (dest)150183: +-------------------------------+150184: ip4_input: p->len 168 p->tot_len 168150184: pbuf_header: old 20004A8E new 20004AA2 (-20)150185: TCP header:150186: +-------------------------------+150186: | 10007 | 49154 | (src port, dest port)150187: +-------------------------------+150188: | 0000907989 | (seq no)150188: +-------------------------------+150189: | 0000072406 | (ack no)150189: +-------------------------------+150190: | 5 | |011000| 1536 | (hdrlen, flags (150190: PSH150191: ACK150192:150192: ), win)150193: +-------------------------------+150193: | 0x3dac | 0 | (chksum, urgp)150194: +-------------------------------+150195: pbuf_header: old 20004AA2 new 20004AB6 (-20)150195: +-+-+-+-+-+-+-+-+-+-+-+-+-+- tcp_input: flags150196: PSH150197: ACK150197:150198: -+-+-+-+-+-+-+-+-+-+-+-+-+-+150198: State: ESTABLISHED150199: tcp_receive: window update 1536150199: tcp_receive: congestion avoidance cwnd 45404150200: tcp_receive: ACK for 72406, unacked->seqno 72278:72406150201: tcp_receive: removing 72278:72406 from pcb->unacked150202: tcp_receive: queuelen 1 ...150203: pbuf_free(20002BD4)150203: pbuf_free: deallocating 20002BD4150204: 0 (after freeing unacked)150204: tcp_receive: pcb->rttest 275 rtseq 72278 ackno 72406150205: tcp_receive: experienced rtt 0 ticks (0 msec).150206: tcp_receive: RTO 3 (1500 milliseconds)150207: tcp_output: nothing to send (00000000)150207: tcp_output: snd_wnd 1536, cwnd 45404, wnd 1536, seg == NULL, ack 72406150208: State: ESTABLISHED150209: tcp_fasttmr: delayed ACK150209: pbuf_alloc(length=20)150210: pbuf_alloc(length=20) == 20002BD4150210: tcp_output: sending ACK for 908117150211: pbuf_header: old 20002C08 new 20002BF4 (20)150211: ip4_output_if: st0150212: IP header:150212: +-------------------------------+150213: | 4 | 5 | 0x00 | 40 | (v, hl, tos, len)150213: +-------------------------------+150214: | 872 |000| 0 | (id, flags, offset)150214: +-------------------------------+150215: | 255 | 6 | 0x0000 | (ttl, proto, chksum)150215: +-------------------------------+150216: | 192 | 168 | 3 | 64 | (src)150216: +-------------------------------+150217: | 192 | 168 | 3 | 68 | (dest)150217: +-------------------------------+150218: ip4_output_if: call netif->output()150219: pbuf_header: old 20002BF4 new 20002BE6 (14)150219: pbuf_free(20002BD4)150220: pbuf_free: deallocating 20002BD4150177: lwip_recvfrom(0, 100053BC, 100, 0x8, ..)150221: lwip_recvfrom: top while sock->lastdata=00000000150221: lwip_recvfrom(0): returning EWOULDBLOCK150173: netconn_recv_data: received 20005750, len=128150223: lwip_recvfrom: netconn_recv err=0, netbuf=20005750150224: lwip_recvfrom: buflen=128 len=128 off=0 sock->lastoffset=0150224: lwip_recvfrom(2): addr=150225: 192.168.3.68150225: port=10007 len=128150226: lwip_recvfrom: deleting netbuf=20005750150226: pbuf_free(20005750)150227: pbuf_free: deallocating 20005750150227: lwip_send(2, data=20010088, size=104, flags=0x0)150228: tcp_write(pcb=20008E00, data=20010088, len=104, apiflags=1)150228: tcp_write: queuelen: 0150229: pbuf_alloc(length=104)150230: pbuf_alloc(length=104) == 20002BD4150230: pbuf_header: old 20002C1C new 20002C08 (20)150231: tcp_write: queueing 72406:72510150232: tcp_write: 1 (after enqueued)150233: tcp_output: snd_wnd 1536, cwnd 45404, wnd 1536, effwnd 104, seq 72406, ack 72406150233: tcp_output: snd_wnd 1536, cwnd 45404, wnd 1536, effwnd 104, seq 72406, ack 72406, i 0150234: tcp_output_segment: rtseq 72406150235: tcp_output_segment: 72406:72510150236: pbuf_header: old 20002C08 new 20002BF4 (20)150236: ip4_output_if: st0150237: IP header:150238: +-------------------------------+150238: | 4 | 5 | 0x00 | 144 | (v, hl, tos, len)150239: +-------------------------------+150240: | 873 |000| 0 | (id, flags, offset)150240: +-------------------------------+150241: | 255 | 6 | 0x0000 | (ttl, proto, chksum)150241: +-------------------------------+150242: | 192 | 168 | 3 | 64 | (src)150242: +-------------------------------+150243: | 192 | 168 | 3 | 68 | (dest)150243: +-------------------------------+150244: ip4_output_if: call netif->output()150245: pbuf_header: old 20002BF4 new 20002BE6 (14)150242: lwip_recvfrom(0, 100053BC, 100, 0x8, ..)150246: lwip_recvfrom: top while sock->lastdata=00000000150246: lwip_recvfrom(0): returning EWOULDBLOCK150245: lwip_send(2) err=0 written=104150255: pbuf_alloc(length=60)150255: pbuf_alloc: allocated pbuf 20005750150255: pbuf_alloc(length=60) == 20005750150256: tcpip_thread: PACKET 20006904150256: pbuf_header: old 20005760 new 2000576E (-14)150257: ip_input: iphdr->dest 0x4003a8c0 netif->ip_addr 0x4003a8c0 (0x3a8c0, 0x3a8c0, 0x40000000)150257: ip4_input: packet accepted on interface st150258: ip4_input:150259: IP header:150260: +-------------------------------+150260: | 4 | 5 | 0x00 | 40 | (v, hl, tos, len)150261: +-------------------------------+150262: | 2207 |000| 0 | (id, flags, offset)150262: +-------------------------------+150263: | 255 | 6 | 0x2b5c | (ttl, proto, chksum)150263: +-------------------------------+150264: | 192 | 168 | 3 | 68 | (src)150264: +-------------------------------+150265: | 192 | 168 | 3 | 64 | (dest)150265: +-------------------------------+150266: ip4_input: p->len 40 p->tot_len 40150267: pbuf_header: old 2000576E new 20005782 (-20)150267: TCP header:150268: +-------------------------------+150268: | 10007 | 49154 | (src port, dest port)150269: +-------------------------------+150270: | 0000908117 | (seq no)150270: +-------------------------------+150271: | 0000072510 | (ack no)150271: +-------------------------------+150272: | 5 | |010000| 2920 | (hdrlen, flags (150272: ACK150273:150273: ), win)150274: +-------------------------------+150274: | 0x3edc | 0 | (chksum, urgp)150275: +-------------------------------+150276: pbuf_header: old 20005782 new 20005796 (-20)150276: +-+-+-+-+-+-+-+-+-+-+-+-+-+- tcp_input: flags150277: ACK150278:150279: -+-+-+-+-+-+-+-+-+-+-+-+-+-+150279: State: ESTABLISHED150280: tcp_receive: window update 2920150280: tcp_receive: congestion avoidance cwnd 45450150281: tcp_receive: ACK for 72510, unacked->seqno 72406:72510150281: tcp_receive: removing 72406:72510 from pcb->unacked150282: tcp_receive: queuelen 1 ...150283: pbuf_free(20002BD4)150283: pbuf_free: deallocating 20002BD4150284: 0 (after freeing unacked)150284: tcp_receive: pcb->rttest 275 rtseq 72406 ackno 72510150285: tcp_receive: experienced rtt 0 ticks (0 msec).150286: tcp_receive: RTO 3 (1500 milliseconds)150287: tcp_output: nothing to send (00000000)150287: tcp_output: snd_wnd 2920, cwnd 45450, wnd 2920, seg == NULL, ack 72510150288: State: ESTABLISHED150289: pbuf_free(20005750)150289: pbuf_free: deallocating 20005750150288: pbuf_alloc(length=158)150291: pbuf_alloc: allocated pbuf 20005750150291: pbuf_alloc(length=158) == 20005750150292: tcpip_thread: PACKET 20006904150292: pbuf_header: old 20005760 new 2000576E (-14)150293: ip_input: iphdr->dest 0x4003a8c0 netif->ip_addr 0x4003a8c0 (0x3a8c0, 0x3a8c0, 0x40000000)150293: ip4_input: packet accepted on interface st150294: ip4_input:150295: IP header:150295: +-------------------------------+150296: | 4 | 5 | 0x00 | 144 | (v, hl, tos, len)150296: +-------------------------------+150297: | 2208 |000| 0 | (id, flags, offset)150297: +-------------------------------+150298: | 255 | 6 | 0x2af3 | (ttl, proto, chksum)150298: +-------------------------------+150299: | 192 | 168 | 3 | 68 | (src)150299: +-------------------------------+150300: | 192 | 168 | 3 | 64 | (dest)150300: +-------------------------------+150301: ip4_input: p->len 144 p->tot_len 144150301: pbuf_header: old 2000576E new 20005782 (-20)150302: TCP header:150303: +-------------------------------+150303: | 10007 | 49154 | (src port, dest port)150304: +-------------------------------+150305: | 0000908117 | (seq no)150305: +-------------------------------+150306: | 0000072510 | (ack no)150306: +-------------------------------+150307: | 5 | |011000| 2920 | (hdrlen, flags (150307: PSH150308: ACK150308:150309: ), win)150309: +-------------------------------+150310: | 0x8617 | 0 | (chksum, urgp)150310: +-------------------------------+150311: pbuf_header: old 20005782 new 20005796 (-20)150311: +-+-+-+-+-+-+-+-+-+-+-+-+-+- tcp_input: flags150312: PSH150313: ACK150313:150314: -+-+-+-+-+-+-+-+-+-+-+-+-+-+150314: State: ESTABLISHED150315: tcp_receive: pcb->rttest 0 rtseq 72406 ackno 72510150315: tcp_output: nothing to send (00000000)150316: tcp_output: snd_wnd 2920, cwnd 45450, wnd 2920, seg == NULL, ack 72510150316: State: ESTABLISHED150267: lwip_recvfrom(0, 100053BC, 100, 0x8, ..)150318: lwip_recvfrom: top while sock->lastdata=00000000150318: lwip_recvfrom(0): returning EWOULDBLOCK150258: lwip_recvfrom(1, 20010088, 128, 0x0, ..)150320: lwip_recvfrom: top while sock->lastdata=00000000150321: lwip_recvfrom(1): returning EWOULDBLOCK150322: lwip_recvfrom(2, 20010088, 128, 0x0, ..)150322: lwip_recvfrom: top while sock->lastdata=00000000150323: tcp_recved: received 128 bytes, wnd 2816 (104).150323: netconn_recv_data: received 20004A70, len=128150324: lwip_recvfrom: netconn_recv err=0, netbuf=20004A70150324: lwip_recvfrom: buflen=128 len=128 off=0 sock->lastoffset=0150325: lwip_recvfrom(2): addr=150326: 192.168.3.68150326: port=10007 len=128150327: lwip_recvfrom: deleting netbuf=20004A70150327: pbuf_free(20004A70)150328: pbuf_free: deallocating 20004A70150338: lwip_recvfrom(1, 20010088, 128, 0x0, ..)150338: lwip_recvfrom: top while sock->lastdata=00000000150339: lwip_recvfrom(0, 100053BC, 100, 0x8, ..)150340: lwip_recvfrom: top while sock->lastdata=00000000150340: lwip_recvfrom(0): returning EWOULDBLOCK150339: lwip_recvfrom(1): returning EWOULDBLOCK150342: lwip_recvfrom(2, 20010088, 128, 0x0, ..)150342: lwip_recvfrom: top while sock->lastdata=00000000150343: pbuf_alloc(length=20)150344: pbuf_alloc(length=20) == 20002BD4150344: tcp_output: sending ACK for 908221150345: pbuf_header: old 20002C08 new 20002BF4 (20)150345: ip4_output_if: st0150346: IP header:150347: +-------------------------------+150347: | 4 | 5 | 0x00 | 40 | (v, hl, tos, len)150348: +-------------------------------+150349: | 874 |000| 0 | (id, flags, offset)150349: +-------------------------------+150350: | 255 | 6 | 0x0000 | (ttl, proto, chksum)150350: +-------------------------------+150351: | 192 | 168 | 3 | 64 | (src)150351: +-------------------------------+150352: | 192 | 168 | 3 | 68 | (dest)150352: +-------------------------------+150353: ip4_output_if: call netif->output()150354: pbuf_header: old 20002BF4 new 20002BE6 (14)150354: pbuf_free(20002BD4)150355: pbuf_free: deallocating 20002BD4150355: tcp_recved: received 104 bytes, wnd 2920 (0).150356: netconn_recv_data: received 20005750, len=104150356: lwip_recvfrom: netconn_recv err=0, netbuf=20005750150357: lwip_recvfrom: buflen=104 len=128 off=0 sock->lastoffset=0150358: lwip_recvfrom(2): addr=150359: 192.168.3.68150359: port=10007 len=104150360: lwip_recvfrom: deleting netbuf=20005750150360: pbuf_free(20005750)150361: lwip_recvfrom(0, 100053BC, 100, 0x8, ..)150362: lwip_recvfrom: top while sock->lastdata=00000000150362: lwip_recvfrom(0): returning EWOULDBLOCK150361: pbuf_free: deallocating 20005750150374: lwip_recvfrom(1, 20010088, 128, 0x0, ..)150374: lwip_recvfrom: top while sock->lastdata=00000000150374: lwip_recvfrom(1): returning EWOULDBLOCK150375: lwip_recvfrom(2, 20010088, 128, 0x0, ..)150375: lwip_recvfrom: top while sock->lastdata=00000000150376: lwip_recvfrom(2): returning EWOULDBLOCK150383: lwip_recvfrom(0, 100053BC, 100, 0x8, ..)150383: lwip_recvfrom: top while sock->lastdata=00000000150384: lwip_recvfrom(0): returning EWOULDBLOCKI noticed e.g.tcp_receive: congestion avoidance cwnd 45404which googles to interesting stuff but way above my pay grade. Like thishttps://lists.gnu.org/archive/html/lwip-users/2017-08/msg00043.htmlMy lwipopts.h is below. I spent many days tweaking different parameters and determining experimentally how much RAM each option uses upCode:/********************************************************************************* @file LwIP/LwIP_HTTP_Server_Netconn_RTOS/Inc/lwipopts.h* @author MCD Application Team* @brief lwIP Options Configuration.********************************************************************************This sort of explains the memory usage*https://lwip-users.nongnu.narkive.com/dkzkPa8l/lwip-memory-settings*https://www.cnblogs.com/shangdawei/p/3494148.html*https://lwip.fandom.com/wiki/Tuning_TCP*https://groups.google.com/g/osdeve_mirror_tcpip_lwip/c/lFYJ7Fw0Cxg*ST UM1713 document gives an overview of integrating all this.*****7/7/22PHMEM_SIZE set to 5k (was 10k). Only ~1.5k is used.*13/7/22PHMEMP_MEM_MALLOC=1, MEM_SIZE=16k.*14/7/22PHMEMP_MEM_MALLOC=0; 1 was unreliable. 8 x 512 byte buffers now.*6/8/22PH4 x MTU buffers for RX. Done partly for EditGetData().*MEM_SIZE=6k. 5k is used - see .map file for ram_heap address.*Sizes of various static RAM structures determined experimentally.* 17/12/22PHLWIP_TCP_KEEPALIVE=1 (for ethser)********/#ifndef __LWIPOPTS_H__#define __LWIPOPTS_H__/*** NO_SYS==1: Provides VERY minimal functionality. Otherwise,* use lwIP facilities.*/#define NO_SYS0// Flag to make LWIP API thread-safe. The netconn and socket APIs are claimed// to be thread-safe anyway. The raw API is never thread-safe.#define LWIP_TCPIP_CORE_LOCKING1// This places more objects into the static block defined by MEM_SIZE.// Uses mem_malloc/mem_free instead of the lwip pool allocator.// MEM_SIZE now needs to be increased by about 10k.// It doesn't magically produce extra memory, and causes crashes.// There is also a performance loss, apparently. AVOID.#define MEMP_MEM_MALLOC0//NC: Need for sending PING messages by keepalive#define LWIP_RAW1#define DEFAULT_RAW_RECVMBOX_SIZE4// For ETHSER#define LWIP_TCP_KEEPALIVE1/*-----------------------------------------------------------------------------*//* LwIP Stack Parameters (modified compared to initialization value in opt.h) -*//* Parameters set in STM32CubeMX LwIP Configuration GUI -*//*----- Value in opt.h for LWIP_DNS: 0 -----*/#define LWIP_DNS 1/* ---------- Memory options ---------- *//* MEM_ALIGNMENT: should be set to the alignment of the CPU for whichlwIP is compiled. 4 byte alignment -> define MEM_ALIGNMENT to 4, 2byte alignment -> define MEM_ALIGNMENT to 2. */#define MEM_ALIGNMENT 4// MEM_SIZE: the size of the heap memory. This is a statically allocated block. You can find it// in the .map file as the symbol ram_heap and you can see how much of this RAM gets used.// If MEMP_MEM_MALLOC=0, this holds just the PBUF_ stuff.// If MEMP_MEM_MALLOC=1 (which is not reliable) this greatly expands and needs 16k+.// Empirically this needs to be big enough for at least 4 x PBUF_POOL_BUFSIZE.// 6k yields a good speed and going to 8k+ makes a minimal improvement. The main// factor affecting speed is the poll period in ethernetif_input().// This value also limits the biggest block size sent out by netconn_write. With the// MEM_SIZE of 6k, the biggest block netconn_write will send out is 4k.#define MEM_SIZE (6*1024)// MEMP_ structures. Their sizes have been determined experimentally, by// increasing them and seeing free RAM changing./* MEMP_NUM_PBUF: the number of memp struct pbufs. If the applicationsends a lot of data out of ROM (or other static memory), thisshould be set high. *///NC: Increased to 20 for ethser#define MEMP_NUM_PBUF 20// each 1 is 20 bytes of static RAM/* MEMP_NUM_UDP_PCB: the number of UDP protocol control blocks. Oneper active UDP "connection". */#define MEMP_NUM_UDP_PCB 6// each 1 is 32 bytes of static RAM/* MEMP_NUM_TCP_PCB: the number of simultaneously active TCPconnections. *///NC: Increased to 20 for ethser#define MEMP_NUM_TCP_PCB 20// each 1 is 145 bytes of static RAM//NC: Have more sockets available. Is set to 4 in opt.h#define MEMP_NUM_NETCONN10/* MEMP_NUM_TCP_PCB_LISTEN: the number of listening TCPconnections. *///NC: Increased to 20 for ethser#define MEMP_NUM_TCP_PCB_LISTEN 20// each 1 is 28 bytes of static RAM/* MEMP_NUM_TCP_SEG: the number of simultaneously queued TCPsegments. */// Was 8; increased to 16 as it improves ETHSER reliability when running// HTTP server#define MEMP_NUM_TCP_SEG 16 // each 1 is 20 bytes of static RAM/* MEMP_NUM_SYS_TIMEOUT: the number of simulateously activetimeouts. */#define MEMP_NUM_SYS_TIMEOUT 10// each 1 is 16 bytes of static RAM/* ---------- Pbuf dynamically allocated buffer blocks ---------- */// These settings are probably for RECEIVE only and make negligible difference// to performance, which is dominated by the low_level_input poll period. These// PBUFs relate directly to the netconn API netbufs etc./* PBUF_POOL_SIZE: the number of buffers in the pbuf pool. */// Statically allocated, so setting this to 2 frees up another 3k of RAM. But any// value under 4 slows down the data rate a lot.#define PBUF_POOL_SIZE 4/* PBUF_POOL_BUFSIZE: the size of each pbuf in the pbuf pool. */// ** It is better to use 4 x MTU than 8 x 512 because in say EditGetData() you are **// ** much more likely to get the whole file header in the first packet. With 512 **// ** byte packets the CRLFCRLF marker is only ~64 bytes before the end of the pkt **// ** Same issue in UploadGetData where we need to get the whole header in. **#define PBUF_POOL_BUFSIZE 1500 + PBUF_LINK_ENCAPSULATION_HLEN + PBUF_LINK_HLEN + PBUF_IP_HLEN + PBUF_TRANSPORT_HLEN/* --------------------------------------------------------------- *//* ---------- TCP options ---------- */#define LWIP_TCP 1#define TCP_TTL 255/* Controls if TCP should queue segments that arrive out oforder. Define to 0 if your device is low on memory. */#define TCP_QUEUE_OOSEQ 0/* TCP Maximum segment size. */#define TCP_MSS (1500 - 40)/* TCP_MSS = (Ethernet MTU - IP header size - TCP header size) *//* TCP sender buffer space (bytes). */#define TCP_SND_BUF (4*TCP_MSS)// no effect on static RAM/* TCP_SND_QUEUELEN: TCP sender buffer space (pbufs). This must be at leastas much as (2 * TCP_SND_BUF/TCP_MSS) for things to work. */// Was 2*; increased to 4* as it improves ETHSER reliability when running// HTTP server#define TCP_SND_QUEUELEN (4* TCP_SND_BUF/TCP_MSS) // (2* TCP_SND_BUF/TCP_MSS)/* TCP advertised receive window. */// Should be less than PBUF_POOL_SIZE * (PBUF_POOL_BUFSIZE - protocol headers)#define TCP_WND (2*TCP_MSS)// no effect on static RAM/* ---------- ICMP options ---------- */#define LWIP_ICMP1/* ---------- DHCP options ---------- */#define LWIP_DHCP 1/* ---------- UDP options ---------- */#define LWIP_UDP 1#define UDP_TTL 255/* ---------- Statistics options ---------- */#define LWIP_STATS 0/* ---------- link callback options ---------- *//* LWIP_NETIF_LINK_CALLBACK==1: Support a callback function from an interface* whenever the link changes (i.e., link down)* 8/2022 this is done from the low_level_input RTOS task.*/#define LWIP_NETIF_LINK_CALLBACK 0/*------------------------------------------------ Checksum options ------------------------------------------------*//*The STM32F4xx allows computing and verifying the IP, UDP, TCP and ICMP checksums by hardware:- To use this feature let the following define uncommented.- To disable it and process by CPU comment the the checksum.*/#define CHECKSUM_BY_HARDWARE#ifdef CHECKSUM_BY_HARDWARE/* CHECKSUM_GEN_IP==0: Generate checksums by hardware for outgoing IP packets.*/#define CHECKSUM_GEN_IP 0/* CHECKSUM_GEN_UDP==0: Generate checksums by hardware for outgoing UDP packets.*/#define CHECKSUM_GEN_UDP 0/* CHECKSUM_GEN_TCP==0: Generate checksums by hardware for outgoing TCP packets.*/#define CHECKSUM_GEN_TCP 0/* CHECKSUM_CHECK_IP==0: Check checksums by hardware for incoming IP packets.*/#define CHECKSUM_CHECK_IP 0/* CHECKSUM_CHECK_UDP==0: Check checksums by hardware for incoming UDP packets.*/#define CHECKSUM_CHECK_UDP 0/* CHECKSUM_CHECK_TCP==0: Check checksums by hardware for incoming TCP packets.*/#define CHECKSUM_CHECK_TCP 0/* CHECKSUM_CHECK_ICMP==0: Check checksums by hardware for incoming ICMP packets.*/#define CHECKSUM_GEN_ICMP 0#else/* CHECKSUM_GEN_IP==1: Generate checksums in software for outgoing IP packets.*/#define CHECKSUM_GEN_IP 1/* CHECKSUM_GEN_UDP==1: Generate checksums in software for outgoing UDP packets.*/#define CHECKSUM_GEN_UDP 1/* CHECKSUM_GEN_TCP==1: Generate checksums in software for outgoing TCP packets.*/#define CHECKSUM_GEN_TCP 1/* CHECKSUM_CHECK_IP==1: Check checksums in software for incoming IP packets.*/#define CHECKSUM_CHECK_IP 1/* CHECKSUM_CHECK_UDP==1: Check checksums in software for incoming UDP packets.*/#define CHECKSUM_CHECK_UDP 1/* CHECKSUM_CHECK_TCP==1: Check checksums in software for incoming TCP packets.*/#define CHECKSUM_CHECK_TCP 1/* CHECKSUM_CHECK_ICMP==1: Check checksums by hardware for incoming ICMP packets.*/#define CHECKSUM_GEN_ICMP 1#endif/*-------------------------------------------------------- Sequential layer options --------------------------------------------------------*//*** LWIP_NETCONN==1: Enable Netconn API (require to use api_lib.c)*/#define LWIP_NETCONN 1/*---------------------------------------------- Socket options ----------------------------------------------*//*** LWIP_SOCKET==1: Enable Socket API (require to use sockets.c)*/#define LWIP_SOCKET 1/*---------------------------------------------- httpd options ----------------------------------------------*//** Set this to 1 to include "fsdata_custom.c" instead of "fsdata.c" for the* file system (to prevent changing the file included in CVS) */#define HTTPD_USE_CUSTOM_FSDATA 0/*------------------------------------------- OS options -------------------------------------------*/#define TCPIP_THREAD_NAME "TCP/IP"#define TCPIP_THREAD_STACKSIZE 4096#define TCPIP_MBOX_SIZE 6#define DEFAULT_UDP_RECVMBOX_SIZE 6#define DEFAULT_TCP_RECVMBOX_SIZE 6#define DEFAULT_ACCEPTMBOX_SIZE 6#define DEFAULT_THREAD_STACKSIZE 512#define TCPIP_THREAD_PRIO osPriorityHigh#define LWIP_DEBUG 1#define IP_DEBUG LWIP_DBG_ON#define DHCP_DEBUG LWIP_DBG_OFF#define UDP_DEBUG LWIP_DBG_ON#define SOCKET_DEBUG_LWIP_DBG_ON//#define ICMP_DEBUG LWIP_DBG_ON|LWIP_DBG_TRACE//#define NETIF_DEBUG LWIP_DBG_OFF#define LWIP_DBG_TYPES_ON (LWIP_DBG_TRACE|LWIP_DBG_STATE)#define LWIP_SO_RCVTIMEO1#define LWIP_NETIF_HOSTNAME1#define SO_REUSE 1// Defining these produces various errors//#define LWIP_IPV61//#define LWIP_IPV6_DHCP61#endif /* __LWIPOPTS_H__ */// TODO#ifdef LWIP_DEBUG#define MEMP_OVERFLOW_CHECK ( 1 )#define MEMP_SANITY_CHECK ( 1 )#define MEM_DEBUG LWIP_DBG_OFF#define MEMP_DEBUG LWIP_DBG_OFF#define PBUF_DEBUG LWIP_DBG_ON#define API_LIB_DEBUG LWIP_DBG_ON#define API_MSG_DEBUG LWIP_DBG_ON#define TCPIP_DEBUG LWIP_DBG_ON#define NETIF_DEBUG LWIP_DBG_ON#define SOCKETS_DEBUG LWIP_DBG_ON#define DEMO_DEBUG LWIP_DBG_ON#define IP_DEBUG LWIP_DBG_ON#define IP6_DEBUG LWIP_DBG_ON#define IP_REASS_DEBUG LWIP_DBG_ON#define RAW_DEBUG LWIP_DBG_ON#define ICMP_DEBUG LWIP_DBG_ON#define UDP_DEBUG LWIP_DBG_ON#define TCP_DEBUG LWIP_DBG_ON#define TCP_INPUT_DEBUG LWIP_DBG_ON#define TCP_OUTPUT_DEBUG LWIP_DBG_ON#define TCP_RTO_DEBUG LWIP_DBG_ON#define TCP_CWND_DEBUG LWIP_DBG_ON#define TCP_WND_DEBUG LWIP_DBG_ON#define TCP_FR_DEBUG LWIP_DBG_ON#define TCP_QLEN_DEBUG LWIP_DBG_ON#define TCP_RST_DEBUG LWIP_DBG_ON#define PPP_DEBUG LWIP_DBG_OFF#define LWIP_DBG_TYPES_ON (LWIP_DBG_ON|LWIP_DBG_TRACE|LWIP_DBG_STATE|LWIP_DBG_FRESH|LWIP_DBG_HALT)#endifThe interface between LWIP and the CPU's ETH subsystem is a fairly standard piece of code which has however been subject to extreme controversy on the ST forum because there have been various bugs, then these got fixed and new bugs arrived, etc. I think the code I have (which a huge amount of time was spent on) is good but there could be bottlenecks.The transmit (low_level_output) i.e. application -> LWIP -> ETH is direct i.e. each packet sent out by LWIP goes straight to the linked list of packets presented to the ETH controller, and this stuff goes out very fast. So fast that any significant buffering is a waste of RAM, I found. I have just 2 buffers in low level ETH.The receive (low_level_input) i.e. ETH -> LWIP -> application (i.e. calling lwip_read and an error code of -1 together with EWOULDBLOCK means there was no rx data) could be done with ETH interrupts but this has been fraught with problems in the ST 32F4 world, so I am using polling. This limits the packet rate to 1 packet every 10ms (which is still easily fast enough for what I need) but the polling loop is switched to 2ms if any data appears, and stays there until a 10ms data gap. It's a kind of simple optimisation, which also prevents an unintentional "DOS" situation (which with interrupts needs to be handled by limiting the interrupt frequency with a timer, etc). So between 100 and 500 packets per second.The serial data is 9600 baud or 115200 baud depending on where along the chain. Yes; this is pretty slow.Sorry for the long post but some say there is not enough detail

author: wek
author_rank: Frequent Contributor
timestamp: January 25, 2023, 11:50:54 am
content: Quote from: peter-h on January 25, 2023, 08:58:00 amIf the network dumps broadcasts at you at a higher rate than 1 packet every 10ms/2ms or whatever, then you'll fail to receive the packets unicast to you.We have had a "funny" situation where a ***-brand printer suddenly felt the urge to broadcast (okay it's multicast) mDNS/LLMNR queries at a cca 10Hz rate cca 2-3 hours every day. Not only the mcu-based gadgets were confused during those periods.JW

author: peter-h
author_rank: Super Contributor
timestamp: January 25, 2023, 12:21:27 pm
content: Right - and this is partly why I (and the Monday guy who did the original ex Cube MX setup and has been working on this on Mondays for some years) decided to not try to implementinterrupt drivenRX (and spend many weeks or months debugging that, with "Piranha" on the ST forum being the only person - apart from you - who knows anything about it, in between telling you how stupid you are, and responding to posts 4 weeks after you've made them) when polling "should" automatically avoid any flooding issues like that, while delivering plenty of speed.There is a significant difference in these gaps between running on the TP-Link switch and running on the Netgear switch.Some tests done uploading datavia TLSto a well known site was achieving a few hundred kbytes/sec which is way more than is needed.But maybe this scheme is causing low level congestion? I can't see how but obviously this stuff is so complex and so hard to debug.To achieve the ST-advertised 95 mbps one obviously need to pull all the tricks out, like sending a fixed data packet using DMA, not going via anything like LWIP. Any real application will be much slower. But I wonder whether something in LWIP is getting into a muddle. I am now running the serial-ETH task in the lowest RTOS priority, so LWIP gets the maximum time to do whatever stuff.I experimented with blocking RTOS task switching around various bits of code and while nothing helped with this issue, I found that one can do a socketwriteduring this (PortEnterCritical etc), a socketreadis blocked until the RTOS has switched to the LWIP task. Perhaps not surprising?Sounds like speeding up the 2ms poll and seeing if it makes any difference might be worth a try, but then one ends up with a potentially hanging target - in possibly many applications. I tried 1ms and it makes no difference.If this really is the problem, it would make sense to implement a crude firewall: an RX ETH ISR which does some very simple rule. I don't know if this is possible, given that one doesn't want to simply drop broadcasts. But one could drop all other packets not addressed to one's IP, surely? OTOH a switch should not be delivering these. So if this is the cause, they must be broadcasts.QuoteWhat is the solution? Even interrupt driven RX must have the IRQ rate limited somehow. And then one will misssomegood packets. In the polled approach one will miss some good packets.

author: ejeffrey
author_rank: Super Contributor
timestamp: January 25, 2023, 04:53:00 pm
content: Quote from: wek on January 25, 2023, 11:50:54 am1 maximum length packet every 2 ms (most broadcast packets are closer to minimum length than maximum) is ~ 6 megabit /s. Even that level of broadcast traffic would cause zero problem for also transferring <1 Mbps of serial data over a 100 megabit network interface.Quote from: peter-h100s of kbyte/second is much faster than the data you expect, but much slower than the network. Especially since you say you are running in polling mode not interrupt driven, you also need to make sure that your logging calls are non-blocking. When you transmit a line of text to the console, how long does it take before you can check the network interface? You need to make that time less than the time it takes for the hardware buffers to fill up. If it takes too long to write the full log message you need to queue the remaining data, go check the network interface, and then continue writing the serial port.QuoteI strongly recommend using a packet sniffer. You should be able to diagnose this with the log data, but the packet sniffer is less invasive, and will also show things that never make it to lwip. You will be able to see if you are getting re-transmits, broadcast packets, and when there is a pause you will be able to see which side transmitted last and figure out which device is failing to respond. You can also use multiple NICs to sniff in different places on your network to see if e.g., different sides of the hub/switch are seeing different data.Can you get collision counters from your network interface? If you are using a hub or a half-duplex connection to a switch you can have collisions, and if something is messed up you can have way more than expected.

author: peter-h
author_rank: Super Contributor
timestamp: January 25, 2023, 05:37:40 pm
content: QuoteIt all runs under FreeRTOS, so "concurrently".On the VCP, there is flow control all the way from the LWIP debug...() function to the Teraterm display and the file logging. This slows down LWIP a bit - probably a factor of two. But the same issues are seen.I think it is most likely internal to LWIP because if I use a different algorithm for the serial -> ETH transfer, the reliability changes a lot. It could be some LWIP buffer config issue, but despite spending a huge amount of time on this originally I never found info I could understand, or anyone who knew about it. There are some experts and I offered them money to look at my LWIP config, without success.The funny thing is that loads of products could have this issue and nobody would notice. It's not the sort of detail most people look at.QuoteI played with them in the past (Etherreal?) but without a lot of knowledge it was of no use.QuoteAccording to thishttps://www.st.com/resource/en/user_manual/um1725-description-of-stm32f4-hal-and-lowlayer-drivers-stmicroelectronics.pdfthere are collision counter registers.Also RM0090 contains a vast number of references, again needing detailed understanding of ETH, but no direct registers for collision counts. There may be software counters. I will dig around.Would a half duplex setting be in the PHY (LAN8742) config? There is no mention of "duplex" in the whole project.EDIT: the duplex config is indeed in the PHY:Code:/* Section 2: PHY configuration section */#define LAN8720A_PHY_ADDRESS0x00//#define DP83848_PHY_ADDRESS 0x01/* PHY Reset delay these values are based on a 1 ms Systick interrupt*/#define PHY_RESET_DELAY (0x000000FFU)/* PHY Configuration delay */#define PHY_CONFIG_DELAY (0x00000FFFU)#define PHY_READ_TO (0x0000FFFFU)#define PHY_WRITE_TO (0x0000FFFFU)/* Section 3: Common PHY Registers */#define PHY_BCR ((uint16_t)0x00) /*!< Transceiver Basic Control Register */#define PHY_BSR ((uint16_t)0x01) /*!< Transceiver Basic Status Register */#define PHY_RESET ((uint16_t)0x8000) /*!< PHY Reset */#define PHY_LOOPBACK ((uint16_t)0x4000) /*!< Select loop-back mode */#define PHY_FULLDUPLEX_100M ((uint16_t)0x2100) /*!< Set the full-duplex mode at 100 Mb/s */#define PHY_HALFDUPLEX_100M ((uint16_t)0x2000) /*!< Set the half-duplex mode at 100 Mb/s */#define PHY_FULLDUPLEX_10M ((uint16_t)0x0100) /*!< Set the full-duplex mode at 10 Mb/s */#define PHY_HALFDUPLEX_10M ((uint16_t)0x0000) /*!< Set the half-duplex mode at 10 Mb/s */#define PHY_AUTONEGOTIATION ((uint16_t)0x1000) /*!< Enable auto-negotiation function */#define PHY_RESTART_AUTONEGOTIATION ((uint16_t)0x0200) /*!< Restart auto-negotiation function */#define PHY_POWERDOWN ((uint16_t)0x0800) /*!< Select the power down mode */#define PHY_ISOLATE ((uint16_t)0x0400) /*!< Isolate PHY from MII */#define PHY_AUTONEGO_COMPLETE ((uint16_t)0x0020) /*!< Auto-Negotiation process completed */#define PHY_LINKED_STATUS ((uint16_t)0x0004) /*!< Valid link established */#define PHY_JABBER_DETECTION ((uint16_t)0x0002) /*!< Jabber condition detected *//* Section 4: Extended PHY Registers *///#define PHY_SR ((uint16_t)0x10) /*!< PHY status register Offset */#define PHY_SR((uint16_t)31)// LAN8270 PHY SR#define PHY_MICR ((uint16_t)0x11) /*!< MII Interrupt Control Register */#define PHY_MISR ((uint16_t)0x12) /*!< MII Interrupt Status and Misc. Control Register */#define PHY_LINK_STATUS ((uint16_t)0x0001) /*!< PHY Link mask *///#define PHY_SPEED_STATUS ((uint16_t)0x0002) /*!< PHY Speed mask */#define PHY_SPEED_STATUS ((uint16_t)0x0004) /*!< PHY Speed mask LAN8270 (set if 10M clear if 100M) *///#define PHY_DUPLEX_STATUS ((uint16_t)0x0004) /*!< PHY Duplex mask */#define PHY_DUPLEX_STATUS ((uint16_t)0x0010) /*!< PHY Duplex mask LAN8270 */#define PHY_MICR_INT_EN ((uint16_t)0x0002) /*!< PHY Enable interrupts */#define PHY_MICR_INT_OE ((uint16_t)0x0001) /*!< PHY Enable output interrupt events */#define PHY_MISR_LINK_INT_EN ((uint16_t)0x0020U) /*!< Enable Interrupt on change of link status */#define PHY_LINK_INTERRUPT ((uint16_t)0x2000U) /*!< PHY link status interrupt mask */

author: ejeffrey
author_rank: Super Contributor
timestamp: January 25, 2023, 07:54:25 pm
content: Wireshark is the "new" (for the last 15 years) name of ethereal, changed due to some legal / trademark ownership issues.It will set the target interface to view every packet received (rather than only ones that match the MAC address), and display each packet, optionally only those that match a particular filter.You can see an example of what a capture looks like here:https://www.wireshark.org/docs/wsug_html_chunked/ChUseMainWindowSection.html#ChUseFig01For each packet you can see the headers and the data, and wireshark dissects most common protocols. It will even automatically highlight some potential problems in red. You will be able to see the data packet lengths, when they are acknowledged, if the same data is transmitted multiple times, if there are broadcast requests like ARP if they are responded to or ignored, and more.Here is a blog post on understanding the phases of a TCP session, using wireshark screen captures at each step:https://packetlife.net/blog/2010/jun/7/understanding-tcp-sequence-acknowledgment-numbers/

author: dare
author_rank: Contributor
timestamp: January 25, 2023, 07:59:09 pm
content: Quote from: peter-h on January 25, 2023, 08:58:00 amRegarding this log snippet: You mention the period lasted a few seconds, however the log only covers 215ms. Is the rest of the log available?Looking at the given log, there is no obvious sign of communication issues: Over the course of the log, the local system (192.168.3 .64) receives two packets with payload data (128 and 104 bytes) from the peer (192.168.3.68), which it ACKs and passes to the application. The application accepts this data via multiple calls to lwip_recvfrom(). During this same time, the local application calls lwip_send() once which results in one packet with payload data (104 bytes) being sent to the peer. An ACK is received from the peer for this packet a short time later. After this exchange of data there is a short period (20ms) of no activity before the log ends. During this period, the local system makes no calls to lwip_send() and no packets are received from the peer.There is no indication of retransmissions in the log, and the TCP send and receive windows are open during the whole exchange. It is possible that the final ACK from the local host to the peer was lost, and that a retransmisson happened after the end of the log. However a single lost packet at this point should not have blocked the peer from sending more data, as the local system's window was open. Without further evidence, I can't speculate why communication halted.

author: peter-h
author_rank: Super Contributor
timestamp: January 25, 2023, 08:47:48 pm
content: All your help guys has given me clues and I think I found it.Code:/* Definition of the Ethernet driver buffers size and count */// 21/7/22 PH increasing both from 2 to 4 makes zero difference to tx speed, and funnily// enough 1 slightly improves it! But only if the low level poll time is 10ms instead of// 20ms; then going to 1 halves the speed. Going to 1+1 yields an extra 3k of RAM.// BUT ETH_RXBUFNB=2 causes multi-second lockups in LWIP due to broadcast packets! 3 fixes// this, so we use 4 to get a margin.#define ETH_RX_BUF_SIZE ETH_MAX_PACKET_SIZE /* buffer size for receive */#define ETH_TX_BUF_SIZE ETH_MAX_PACKET_SIZE /* buffer size for transmit */#define ETH_RXBUFNB 4 // was (4U) /* Rx buffers of size ETH_RX_BUF_SIZE */#define ETH_TXBUFNB 2 // was (4U) /* Tx buffers of size ETH_TX_BUF_SIZE */The original ST Cube MX values were rx=4 and tx=4.In my quest to reduce the fairly wild LWIP memory usage I spent a lot of time and - to the extent that little of how LWIP actually works is documented - chopped the usage down by quite a bit, with no apparent performance penalty.The above ETH buffers are below LWIP; the 32F417 ETH controller is fed via them. I quickly found that on the TX side the data vanishes down the wire so fast that even 1 buffer achieves megabytes/sec, so I went from 4 to 2. On the RX side we are polling (every 10ms, with this dropping to 2ms on any rx data, until 100ms of no data and then it goes back to 10ms) and some more buffering would be expected to be needed. I found 2 is perfectly good enough, though more might be needed on slow WANs etc (4G).But there is a problem there, if "other packets" arrive.I waggled a GPIO every time around that 2ms/10ms loop, and while most of the 2ms blocks corresponded with known data, some didn't, and were obviously kicked off by broadcast packets.And sure enough these mystery packets were kicking off the seconds-long delays in LWIP. They caused some deadlock. Perhaps LWIP is doing something with these but for sure there is no application (on a socket) reading them out.The red trace is the GPIO waggle.I increased the ETH_RXBUFNB back to 4 and it completely solved the problem. 3 also does it, so I went to 4 to get a margin.I reckon, years ago, somebody at ST who was porting LWIP to the 32F4 found that they needed 4 and didn't document why. Or maybe they just started with 4 and got lucky.It kind of makes sense that the more "sessions" you want to run, the more buffers you might need, but should LWIP actually lock up? Anyway these are low level buffers - there is a list of pointers which the 32F4's ETH DMA controller picks up. No software involved as such beyond memcpy to/from the buffer(s) (could use DMA) and setting up some pointer. The ETH DMA controller is continually polling those pointers to see if there is anything to send, and similarly it sets them up if something has arrived. There is "something" in LWIP, combined with the way the 32F4 ETH controller drives that buffer list, which doesn't workreliablywith just 2 rx buffers. Which is weird since that is a raw physical packet level, nothing to do with sessions, sockets, connections...I had already tried all the LWIP buffer settings in lwipopts.h and nothing made any difference.

author: wek
author_rank: Frequent Contributor
timestamp: January 25, 2023, 09:05:36 pm
content: > If the network dumps broadcasts at you at a higher rate than 1 packet> every 10ms/2ms or whatever, then> *you'll fail to receive the packets unicast to you*.So, if your buffer count was 2, that's 1 for the spurious broadcast and 1 for the unicast packet you expect. One more broadcast packet before the unicast packet within given poll time, and you've lost the unicast data. Sure, TCP will recover if it's not too often, but that will cost you exactly the delay you observe.3 or 4 are just arbitrary values "working" in your arbitrary network. A malicious out just runaway device on the network can flood it with short but numerous broadcasts. It's up to you how perfect solution do you want to develop.This is not to say there may be no other mechanisms how the observed problem can occur.JW

author: peter-h
author_rank: Super Contributor
timestamp: January 25, 2023, 09:16:04 pm
content: QuoteA few seconds? I hope that timer is configurableQuoteIt can't be that simple, because there is only one RJ45 socketThere is only one PHY chip. That and the buffers are the physical layer. TCP/IP can have almost any number of sessions (open sockets).The upper limit on sockets is actually configurable. I have about 5 applications running, 2 running TLS, concurrently. And all this works with 2 buffers.Even if you had interrupt-driven RX you could still get two packets arriving within the exec time of the ISR. There is no documented issue like that, that the ISR exec time has to be shorter than the shortest possible 100mbps packet.If the explanation was this simple, ETH would work pretty poorly, because you will always risk congestion of the PHY buffers unless you have enough of them for the worst possible scenario of 100mbps (or 1000mbps), and shortest possible packets. It would be a lot of buffering.I reckon it is some subtle deadlock, because I was getting the multi-second delays (very rarely) even on the isolated switch, withnobroadcasts arriving.

author: dare
author_rank: Contributor
timestamp: January 25, 2023, 09:44:14 pm
content: Quote from: peter-h on January 25, 2023, 08:47:48 pmThere's no "lock-up" in LwIP. If the lack of receive buffers at the network interface level is causing the receiver to drop packets, then its up to the sender to discover this and retransmit them. This takes time, which is resulting in a visible hiccup in throughput. But at no point is the system unable to make forward progress. This is standard behavior for TCP (though tuning can improve things).(Incidentally, a packet trace in Wireshark would have revealed this scenario rather quickly).A simple solution to this problem is to apply packet filtering at the network interface level. It is relatively easy to peek inside the IP header and detect broadcast/multicast packets vs unicast packets. Using this information you can give preference to unicast packets in the receive queue. Simply increment a (thread-safe) counter whenever a broadcast packet is queued, and decrement it whenever it is dequeued. Upon receiving a broadcast packet, if the counter is above a threshold, simply discard the packet rather than queuing it. The threshold can be as low as 1, since broadcast is inherently unreliable, and all broadcast protocols incorporate some form of retransmission. I have shipped a number of LwIP-based consumer products that incorporate this strategy to good effect.

author: dare
author_rank: Contributor
timestamp: January 25, 2023, 09:52:34 pm
content: QuoteJust to re-iterate, I have shipped literally millions of consumer IoT devices based on LwIP. If the system is stumbling or locking up on an isolated network, then there is either a hardware problem or a system configuration problem. Setup correctly, LwIP is very reliable.

author: peter-h
author_rank: Super Contributor
timestamp: January 25, 2023, 10:40:45 pm
content: Thank you, but you are talking in generalities, whereas I have to get specifics to work.QuoteCould you post some code, please? AIUI the ETH RX buffer is filled from the dedicated ETH controller which the 32F417 has, so there is no opportunity to intercept that data before that buffer is filled. What CPU are you using?I mentioned this strategy (a crude "firewall") further back, adding that I never found any implementation of it.For sure one could implement it in low_level_input. In my case, this is a dedicated RTOS task, which AFAICT copies the data into some "PBUF" queue owned by LWIP. That task could be an ISR instead but again I never found good working code example for that, and an RTOS task is a handy solution because it offers natural DOS protection, while enabling the running of things like link status change detection at the same level.Perhapswekwill know more detail.However, as I wrote further back, more buffers will only postpone the vulnerability. AFAICT, unless the ETH hardware and the entire stack above it can handle the physical speed (10Mbytes/sec in this case) the system must be capable of dropping packets, unless it has infinite buffering.I did a search for TCP/IP retransmission and if running on a fast LAN (trip time ~1ms) the initial retry delay should not be what I am seeing, which is 1-2 secs. So I think those delays are something else - unless LWIP does something else.This posthttps://www.eevblog.com/forum/microcontrollers/lwip-tcp_timer-interval/suggests LWIP uses 3 seconds but that is a lot longer than I normally see.This post (whose date shows how old LWIP is)https://lists.nongnu.org/archive/html/lwip-devel/2007-09/msg00061.htmlhas some info suggesting 3 seconds too.But in my project, TCP_TMR_INTERVAL = 250ms.

author: wek
author_rank: Frequent Contributor
timestamp: January 26, 2023, 07:48:38 am
content: If you think it's something else - and as I've said, it may quite well be - feel free to investigate.As I've said, I've positively determined broadcasts and described consequent data drop/TCP repair to be cause of "jerks" in my case and I reckon it's quite common in various SOHO LANs. (The associated act of sniffing the LAN was an eye opener, that's why you are recommended to do this by many here.) We then decided that this is something we can/should solve organisationally rather than striving for perfection at our side. As the LANs are an extremely variable environment completely out of our control, and as there may be varied other factors involved (e.g. wireless hops), we chose to defer the problem to the clients ("use a switch with nothing else on it, works as expected? then clean up your LAN, other devices will benefit from this too"). I don't say this is an universally applicable approach, just that there are many ways to skin a cat, some of which don't require messing with lwip/ETH more than usually.JW

author: peter-h
author_rank: Super Contributor
timestamp: January 26, 2023, 09:25:26 am
content: I see what must be broadcasts (well, whatever gets routed through a switch) roughly every 1-2 seconds.Inevitably, some of these will end up in the ETH RX buffer list at the same time as a real data packet.This is OK if there are 3+ RX buffers (each buffer is MTU-size btw) but it was the 2 buffer case which was comprehensively breaking stuff.I'd like to understand the mechanism for this problem, because those RX buffers are at the PHY level. They all rank equally until they reach LWIP. So two data packets (addressed to the same IP) should have the same effect as 1 data packet and 1 multicast. Is this wrong?Also I noticed that when I opened up a VPN to another site, more multicast packets appeared - as one would probably expect.The issue ofhigh speedmulticasts making a device grind to a halt, I am happy with. It's not possible to guard against this completely while achieving a decent speed - because even an RX ISR will need to implement a rate limit otherwise a stream of multicasts will totally hang up the box. I don't needdareto come back with some sourcecode because the 32F4 can dump broadcasts internally (but that breaks things totally - see below).The Q is then: is there a legitimate scenario where we do want broadcasts to reach LWIP? There is a lot online about this. Thishttps://stackoverflow.com/questions/35130743/how-to-create-a-service-that-sends-receives-udp-broadcasts-on-multiple-interfacesuggests that LWIP passes broadcasts back to every open socket.This suggests that some ST CPUs have a broadcast filter alreadyhttps://stackoverflow.com/questions/50132454/stm32-lwip-mcu-load-due-to-broadcast-packetThis seems quite similar to my problem but he fixed it without ever finding out how - could just be a different # of rx buffers (the H7 has/had loads of ETH issues)https://stackoverflow.com/questions/72323318/lwip-netconn-recv-netbuf-overwritten-from-browser-broadcast-messageBased on thishttps://community.st.com/s/question/0D50X0000ALu7XqSQJ/how-configure-lwip-to-receive-udp-broadcast-using-stm32cubemx-I've done this, in case it helps to dump broadcasts earlier// Disable broadcasts being received#define IP_SOF_BROADCAST 1#define IP_SOF_BROADCAST_RECV 1Everything still works, and indeed I don't have any LWIP application receiving broadcasts.I cannot find anything re LWIPitselfneeding broadcasts for anything.As usual we've done this beforehttps://www.eevblog.com/forum/microcontrollers/any-micros-with-eth-and-hardware-incoming-packet-filtering-by-ip/which led me to BFD of ETH_MACFFR:Bit 5 BFD: Broadcast frames disableWhen this bit is set, the address filters filter all incoming broadcast frames.When this bit is reset, the address filters pass all received broadcast frames.and I disabled broadcasts using using ETH_BROADCASTFRAMESRECEPTION_DISABLE in stm32f4xx_hal_eth.c.That breaks things completely. Evidently the 32F4 ETHdoesneed to process broadcasts (for network config?) but LWIP is happy to dump them.So if I was to dump them, they would need to be dumped at the level above the 32F4's ETH controller i.e. in low_level_input, but since that is above the ETH PHY buffers discussed, what would be the point? The real gain of dumping them at the ETH level would be a high level of DOS protection.

author: nctnico
author_rank: Super Contributor
timestamp: January 26, 2023, 10:42:55 am
content: No, you'd need to put a layer between the ethernet driver and LWIP that throws away the broadcast /unicast messages that are not necessary for the device. So the broadcast/unicast messages will still be received but are not fed into LWIP.

author: peter-h
author_rank: Super Contributor
timestamp: January 26, 2023, 11:07:00 am
content: I think that is what I said:QuoteIt would be a neat approach because it could be runtime configurable whereas the lwipopts.h stuff is fixed at compile time (although one could mod all the code which references them...).$100Presumably the BFD of ETH_MACFFR solution is usable only for boxes which are addressed by MAC #Sources for low_level_input:Code:/*** @brief Should allocate a pbuf and transfer the bytes of the incoming* packet from the interface into the pbuf.** @param netif the lwip network interface structure for this ethernetif* @return a pbuf filled with the received packet (including MAC header)* NULL on memory error*/static struct pbuf * low_level_input(struct netif *netif){struct pbuf *p = NULL, *q = NULL;uint16_t len = 0;uint8_t *buffer;__IO ETH_DMADescTypeDef *dmarxdesc;uint32_t bufferoffset = 0;uint32_t payloadoffset = 0;uint32_t byteslefttocopy = 0;uint32_t i=0;/* get received frame */HAL_StatusTypeDef status = IF_HAL_ETH_GetReceivedFrame(&EthHandle);if (status != HAL_OK){return NULL;// Return if no RX data}else{rxactive=true;// set "seen rx data" flag}/* Obtain the size of the packet and put it into the "len" variable. */len = EthHandle.RxFrameInfos.length;buffer = (uint8_t *)EthHandle.RxFrameInfos.buffer;if (len > 0){/* We allocate a pbuf chain of pbufs from the Lwip buffer pool */p = pbuf_alloc(PBUF_RAW, len, PBUF_POOL);}if (p != NULL){dmarxdesc = EthHandle.RxFrameInfos.FSRxDesc;bufferoffset = 0;for(q = p; q != NULL; q = q->next){byteslefttocopy = q->len;payloadoffset = 0;/* Check if the length of bytes to copy in current pbuf is bigger than Rx buffer size */while( (byteslefttocopy + bufferoffset) > ETH_RX_BUF_SIZE ){/* Copy data to pbuf */memcpy( (uint8_t*)((uint8_t*)q->payload + payloadoffset), (uint8_t*)((uint8_t*)buffer + bufferoffset), (ETH_RX_BUF_SIZE - bufferoffset));/* Point to next descriptor */dmarxdesc = (ETH_DMADescTypeDef *)(dmarxdesc->Buffer2NextDescAddr);buffer = (uint8_t *)(dmarxdesc->Buffer1Addr);byteslefttocopy = byteslefttocopy - (ETH_RX_BUF_SIZE - bufferoffset);payloadoffset = payloadoffset + (ETH_RX_BUF_SIZE - bufferoffset);bufferoffset = 0;}/* Copy remaining data in pbuf */memcpy( (uint8_t*)((uint8_t*)q->payload + payloadoffset), (uint8_t*)((uint8_t*)buffer + bufferoffset), byteslefttocopy);bufferoffset = bufferoffset + byteslefttocopy;}}/* Release descriptors to DMA *//* Point to first descriptor */dmarxdesc = EthHandle.RxFrameInfos.FSRxDesc;/* Set Own bit in Rx descriptors: gives the buffers back to DMA */for (i=0; i< EthHandle.RxFrameInfos.SegCount; i++){//__DMB(); - fossil code for the 32F417, apparently.dmarxdesc->Status |= ETH_DMARXDESC_OWN;dmarxdesc = (ETH_DMADescTypeDef *)(dmarxdesc->Buffer2NextDescAddr);}/* Clear Segment_Count */EthHandle.RxFrameInfos.SegCount =0;/* When Rx Buffer unavailable flag is set: clear it and resume reception */if ((EthHandle.Instance->DMASR & ETH_DMASR_RBUS) != (uint32_t)RESET){/* Clear RBUS ETHERNET DMA flag */EthHandle.Instance->DMASR = ETH_DMASR_RBUS;/* Resume DMA reception */EthHandle.Instance->DMARPDR = 0;}return p;}// Timeout for ethernetif_input "activity" status which drops fast poll to slow poll.// FreeRTOS timer needs to have a callback, even if this function is empty.// The other way to check timer status is// if (xTimerIsTimerActive(logout_timer) == pdTRUE) - gives True if not expiredstatic void RXactiveTimerCallback( TimerHandle_t rxactive_timer ){rx_poll_period=ETH_RX_SLOW_POLL_INTERVAL;}/*** This function is the ethernetif_input task. It uses the function low_level_input()* that handles the actual reception of bytes from the network interface.** This is a standalone RTOS task so is a forever loop.* Could be done with interrupts but then we have the risk of hanging the unit with fast input* (unlikely if ETH is via a switch, but you could have a faulty LAN with lots of* broadcasts).**/void ethernetif_input( void * argument ){struct pbuf *p;struct netif *netif = (struct netif *) argument;// Define RX activity timer, for dropping fast poll down to slow pollTimerHandle_t *rxactive_timer = xTimerCreate("ETH RX active timer", pdMS_TO_TICKS(ETH_SLOW_POLL_DELAY), pdFALSE, NULL, RXactiveTimerCallback);// Start "rx active" timerxTimerStart(rxactive_timer, 20);// 20 is just a wait time for timer allocationdo{/*extern void TopLED(bool state);TopLED(true); // waggleGPIOTopLED(false);*/p = low_level_input( netif );// This sets rxactive=true if it sees dataif (p!=NULL){if (netif->input( p, netif) != ERR_OK ){pbuf_free(p);}}if (rxactive){rxactive=false;// Seen rx data - reload timeoutxTimerReset(rxactive_timer, 20);// Reload "rx active" timeout (with ETH_SLOW_POLL_DELAY)// and get osDelay below to run fastrx_poll_period=ETH_RX_FAST_POLL_INTERVAL;}// This has a dramatic effect on ETH speed, both ways (TCP/IP acks all packets)osDelay(rx_poll_period);} while(true);}

author: dare
author_rank: Contributor
timestamp: January 26, 2023, 08:12:03 pm
content: Quote from: peter-h on January 26, 2023, 09:25:26 amARP requires reception of broadcast packets. Without ARP, IPv4 doesn't work.QuoteEvery point in the code where you have a queue that is fed/serviced in different contexts is subject to queue overflow/buffer exhaustion whenever the packet arrival rate is faster than the processing rate. Right now you have two such queues*. As you've described it, the first queue is fed by the ethernet controller and serviced by your ethernetif_input() task. The second queue is inside LwIP and is fed by the call to netif->input() in ethernetif_input() and serviced by the LwIP tcpip task itself.* (Technically there is also a third queue between the LwIP tcpip task and your application task that is calling lwip_recvfrom(). However since your application-level data rate is so low, this queue is unlikely to suffer from congestion.)Ideally there would only be 1 queue on the input side (more on this later). But given that you have two queues, both need to be optimized to deal with extraneous input packets. For the first queue, it seems your only option is to increase the buffer pool size to the point where it is unlikely to be a problem. So far you have found that 4 buffers works, but some stress testing might reveal that you need more.For the second queue you should install a filter that blocks extraneous packets from being forwards to the tcpip task.QuoteWell, I'm going to give it to you anyway.This function will filter out everything except for ARP packets and IPv4 packets destined to a non-broadcast ether address:Code:static int should_accept_ethernet_packet(const uint8_t * pktData, size_t pktLen){const struct eth_hdr * pktHdr = (const struct eth_hdr *)pktData;/* If the packet is too short, reject it. */if (pktLen < sizeof(struct eth_hdr)) {return 0;}/* If the packet is to a broadcast destination... */if (eth_addr_cmp(&pktHdr->dest, &ethbroadcast)) {/* If the packet is a broadcast ARP packet, accept it. */if (ntohs(pktHdr->type) == ETHTYPE_ARP) {return 1;}}/* if the packet is to an IPv4 multicast destination, reject it. */else if (pktHdr->dest.addr == LL_IP4_MULTICAST_ADDR_0 &&pktHdr->dest.addr == LL_IP4_MULTICAST_ADDR_1 &&pktHdr->dest.addr == LL_IP4_MULTICAST_ADDR_2) {return 0;}/* if the packet is an IPv4 packet, accept it. */else if (ntohs(pktHdr->type) == ETHTYPE_IP) {return 1;}/* Reject all other packets */return 0;}Place a call to this function early in your low_level_input() function, before you allocate the pbuf. If the function returns false, simply ignore the packet. Note that, even with this function in place, you may still need to tune the total number of pbufs. This is especially true if you have data flowing both ways in the connection and have big TCP window sizes.Regarding the need for the ethernetif_input() task, I couldn't find anything to suggest that using interrupts for packet reception was unworkable. Is there a concrete description of the problem you encountered?

author: peter-h
author_rank: Super Contributor
timestamp: January 26, 2023, 10:03:42 pm
content: QuoteOK, so theQuoteisnotpreventing that, because everything works. It seems that this just prevents broadcasts coming back via the LWIP API.QuoteI found 3 makes it solid so I am running with 4 (which happens to be the original Cube default). I've written it all up in the design doc for the future. BTW in case people wonder why I write these long posts, it is because a) I want to learn stuff and b) I drop the thread URL into the design docQuoteThat is truly wonderful of you, but I don't know where to call it. I can see several places where packets can possibly be examined, but how to discard them? For the 32F4 ETH controller to see it "gone" one has to clear some pointer. That is probably all, plus not copy it to LWIP.QuoteYou mean inside IF_HAL_ETH_GetReceivedFrame() ?Code:HAL_StatusTypeDef IF_HAL_ETH_GetReceivedFrame(ETH_HandleTypeDef *heth){uint32_t framelength = 0U;/* Check if segment is not owned by DMA *//* if (((heth->RxDesc->Status & ETH_DMARXDESC_OWN) == (uint32_t)RESET) && ((heth->RxDesc->Status & ETH_DMARXDESC_LS) != (uint32_t)RESET)) *///__DMB();if(((heth->RxDesc->Status & ETH_DMARXDESC_OWN) == (uint32_t)RESET)){/* Check if last segment */if(((heth->RxDesc->Status & ETH_DMARXDESC_LS) != (uint32_t)RESET)){/* increment segment count */(heth->RxFrameInfos).SegCount++;/* Check if last segment is first segment: one segment contains the frame */if ((heth->RxFrameInfos).SegCount == 1U){(heth->RxFrameInfos).FSRxDesc =heth->RxDesc;}heth->RxFrameInfos.LSRxDesc = heth->RxDesc;/* Get the Frame Length of the received packet: substruct 4 bytes of the CRC */framelength = (((heth->RxDesc)->Status & ETH_DMARXDESC_FL) >> ETH_DMARXDESC_FRAMELENGTHSHIFT) - 4U;heth->RxFrameInfos.length = framelength;/* Get the address of the buffer start address */heth->RxFrameInfos.buffer = ((heth->RxFrameInfos).FSRxDesc)->Buffer1Addr;/* point to next descriptor */heth->RxDesc = (ETH_DMADescTypeDef*) ((heth->RxDesc)->Buffer2NextDescAddr);/* Return function status */return HAL_OK;}/* Check if first segment */else if((heth->RxDesc->Status & ETH_DMARXDESC_FS) != (uint32_t)RESET){(heth->RxFrameInfos).FSRxDesc = heth->RxDesc;(heth->RxFrameInfos).LSRxDesc = NULL;(heth->RxFrameInfos).SegCount = 1U;/* Point to next descriptor */heth->RxDesc = (ETH_DMADescTypeDef*) (heth->RxDesc->Buffer2NextDescAddr);}/* Check if intermediate segment */else{(heth->RxFrameInfos).SegCount++;/* Point to next descriptor */heth->RxDesc = (ETH_DMADescTypeDef*) (heth->RxDesc->Buffer2NextDescAddr);}}/* Return function status */return HAL_ERROR;}which gets called thus:Not sure where the packet can be found though.QuoteIt has been done but there are various issues and I have not found a working code example. Also it isn't trivial because one needs to do rate limiting i.e. disable further ints and then re-enable them in a timer ISR. Another thing, not in the code I posted for brevity, is that we are doing link status change detection (cable plugged in etc) in that RTOS task, and it conveniently avoids issues with code running in a different RTOS context.Bear in mind all this stuff is totally unsupported. All one finds is code snippets, mostly somebody's work in progress (much of github is WIP) and reports of problems, mostly with no response, or with slagging off of ST

author: dare
author_rank: Contributor
timestamp: January 27, 2023, 12:45:09 am
content: Quote from: peter-h on January 26, 2023, 10:03:42 pmRoughly speaking, yes. These are build flags which enable the support for the SOF_BROADCAST option on raw and UDP PCBs. Nothing to do with whether the lower parts of the stack accept broadcast packets.QuoteAs I mentioned, the place to put it is in low_level_input(). Like so:Code:static struct pbuf * low_level_input(struct netif *netif){struct pbuf *p = NULL, *q = NULL;uint16_t len = 0;uint8_t *buffer;__IO ETH_DMADescTypeDef *dmarxdesc;uint32_t bufferoffset = 0;uint32_t payloadoffset = 0;uint32_t byteslefttocopy = 0;uint32_t i=0;/* get received frame */HAL_StatusTypeDef status = IF_HAL_ETH_GetReceivedFrame(&EthHandle);if (status != HAL_OK){return NULL;// Return if no RX data}else{rxactive=true;// set "seen rx data" flag}/* Obtain the size of the packet and put it into the "len" variable. */len = EthHandle.RxFrameInfos.length;buffer = (uint8_t *)EthHandle.RxFrameInfos.buffer;/***** IF THE PACKET SHOULD BE ACCEPTED... *****/if (should_accept_ethernet_packet(buffer, len)){/* We allocate a pbuf chain of pbufs from the Lwip buffer pool */p = pbuf_alloc(PBUF_RAW, len, PBUF_POOL);}if (p != NULL){dmarxdesc = EthHandle.RxFrameInfos.FSRxDesc;bufferoffset = 0;for(q = p; q != NULL; q = q->next){byteslefttocopy = q->len;payloadoffset = 0;/* Check if the length of bytes to copy in current pbuf is bigger than Rx buffer size */while( (byteslefttocopy + bufferoffset) > ETH_RX_BUF_SIZE ){/* Copy data to pbuf */memcpy( (uint8_t*)((uint8_t*)q->payload + payloadoffset), (uint8_t*)((uint8_t*)buffer + bufferoffset), (ETH_RX_BUF_SIZE - bufferoffset));/* Point to next descriptor */dmarxdesc = (ETH_DMADescTypeDef *)(dmarxdesc->Buffer2NextDescAddr);buffer = (uint8_t *)(dmarxdesc->Buffer1Addr);byteslefttocopy = byteslefttocopy - (ETH_RX_BUF_SIZE - bufferoffset);payloadoffset = payloadoffset + (ETH_RX_BUF_SIZE - bufferoffset);bufferoffset = 0;}/* Copy remaining data in pbuf */memcpy( (uint8_t*)((uint8_t*)q->payload + payloadoffset), (uint8_t*)((uint8_t*)buffer + bufferoffset), byteslefttocopy);bufferoffset = bufferoffset + byteslefttocopy;}}/* Release descriptors to DMA *//* Point to first descriptor */dmarxdesc = EthHandle.RxFrameInfos.FSRxDesc;/* Set Own bit in Rx descriptors: gives the buffers back to DMA */for (i=0; i< EthHandle.RxFrameInfos.SegCount; i++){//__DMB(); - fossil code for the 32F417, apparently.dmarxdesc->Status |= ETH_DMARXDESC_OWN;dmarxdesc = (ETH_DMADescTypeDef *)(dmarxdesc->Buffer2NextDescAddr);}/* Clear Segment_Count */EthHandle.RxFrameInfos.SegCount =0;/* When Rx Buffer unavailable flag is set: clear it and resume reception */if ((EthHandle.Instance->DMASR & ETH_DMASR_RBUS) != (uint32_t)RESET){/* Clear RBUS ETHERNET DMA flag */EthHandle.Instance->DMASR = ETH_DMASR_RBUS;/* Resume DMA reception */EthHandle.Instance->DMARPDR = 0;}return p;}QuoteThis post contains code that is purported to work:https://lwip-users.nongnu.narkive.com/62tlpY0j/stm32-rbus-receive-buffer-unavailable-bit-set-after-debugger-break#post7. The code looks roughly sound to me, especially the logic in ethernetif_input() which drains all the packets in the DMA queue before clearing the Rx Buffer unavailable flag and re-enabling reception.Here is the code from the post in an easier to read form:Code:// from the sample code for the STM32F4x7 ethernet FreeRTOS/lwIP demo./*** Should allocate a pbuf and transfer the bytes of the incoming* packet from the interface into the pbuf.** @param netif the lwip network interface structure for this ethernetif* @return a pbuf filled with the received packet (including MAC header)* NULL on memory error*/static struct pbuf *low_level_input(struct netif *netif){struct pbuf *p, *q;u16_t len;uint32_t l = 0, i = 0;FrameTypeDef frame;u8 *buffer;__IO ETH_DMADESCTypeDef *DMARxNextDesc;p = NULL;/* Get received frame */frame = ETH_Get_Received_Frame_interrupt();if (frame.descriptor && frame.buffer){/* check that frame has no error */if ((frame.descriptor->Status & ETH_DMARxDesc_ES) == (uint32_t)RESET){/* Obtain the size of the packet and put it into the "len"variable. */len = frame.length;buffer = (u8 *)frame.buffer;/* We allocate a pbuf chain of pbufs from the pool. */p = pbuf_alloc(PBUF_RAW, len, PBUF_POOL);/* Copy received frame from ethernet driver buffer to stackbuffer */if (p != NULL){for (q = p; q != NULL; q = q->next){memcpy((u8_t *)q->payload, (u8_t *)&buffer, q->len);l = l + q->len;}}}/* Release descriptors to DMA *//* Check if received frame with multiple DMA buffer segments */if (DMA_RX_FRAME_infos->Seg_Count > 1){DMARxNextDesc = DMA_RX_FRAME_infos->FS_Rx_Desc;}else{DMARxNextDesc = frame.descriptor;}/* Set Own bit in Rx descriptors: gives the buffers back to DMA */for (i = 0; i < DMA_RX_FRAME_infos->Seg_Count; i++){DMARxNextDesc->Status = ETH_DMARxDesc_OWN;DMARxNextDesc = (ETH_DMADESCTypeDef*)(DMARxNextDesc->Buffer2NextDescAddr);}/* Clear Segment_Count */DMA_RX_FRAME_infos->Seg_Count = 0;}return p;}static void ethernet_watchdog(void){/* When Rx Buffer unavailable flag is set: clear it and resumereception */if ((ETH->DMASR & ETH_DMASR_RBUS) != (u32)RESET){/* Clear RBUS ETHERNET DMA flag */ETH->DMASR = ETH_DMASR_RBUS;/* Resume DMA reception. The register doesn't care what youwrite to it. */ETH->DMARPDR = 0;}}void ethernetif_input(void *pvParameters){struct pbuf *p;for (;;){if (xSemaphoreTake(s_xSemaphore,emacBLOCK_TIME_WAITING_FOR_INPUT) == pdTRUE){while ((p = low_level_input(s_pxNetIf)) != 0){if (p != 0){if (ERR_OK != s_pxNetIf->input(p, s_pxNetIf)){pbuf_free(p);p = NULL;}}}}ethernet_watchdog();}}QuoteWhy? The interrupt rate should be no faster than the inbound packet rate, which given a 100Mb link and typical packet sizes isn't all that fast. Plus trains of interrupts are limited by the number of DMA buffers configured. When all the buffers are full, the interrupts stop.QuoteThat can still happen. Just put it inside the receive semaphore wait loop (for loop in the above example code) and set the semaphore wait timeout to an appropriate value.

author: peter-h
author_rank: Super Contributor
timestamp: January 27, 2023, 07:15:34 am
content: Many thanks. I will test that later today. Going to a wedding - yawn.My original polled low level input code did actually loop round until the rx queue was drained (having been checking for data every 10ms) but this was removed because it could be stuck in that loop if stuff was arriving fast. But now that we will be dumping broadcasts, I might put that back in, because at least those packets will not be stuffed into LWIP anymore.

author: peter-h
author_rank: Super Contributor
timestamp: January 28, 2023, 09:33:31 am
content: I don't think the above function is quite right. See test reports below. First, I changed it to a boolean:Code:// Function used to test whether a packet is to be rejected - broadcasts etc// Returns true if packet is to be accepted.static bool should_accept_ethernet_packet(const uint8_t * pktData, size_t pktLen){if (g_eth_multi) return true;const struct eth_hdr * pktHdr = (const struct eth_hdr *)pktData;/* If the packet is too short, reject it. */if (pktLen < sizeof(struct eth_hdr)){return false;}/* If the packet is a broadcast ARP accept it */if ( eth_addr_cmp(&pktHdr->dest, &ethbroadcast) ){if (ntohs(pktHdr->type) == ETHTYPE_ARP){return true;}}// if the packet is to an IPv4 multicast destination, reject it.if (pktHdr->dest.addr == LL_IP4_MULTICAST_ADDR_0 &&pktHdr->dest.addr == LL_IP4_MULTICAST_ADDR_1 &&pktHdr->dest.addr == LL_IP4_MULTICAST_ADDR_2){return false;}/* if the packet is an IPv4 packet, accept it. */if (ntohs(pktHdr->type) == ETHTYPE_IP){return true;}/* Reject all other packets WRONG TODO */return false;}Hard to debug because there are so many packets and I don't know the format details. But someone may spot something obvious.This test is taken at about the right time for where I would expect data packetsCode:/* if the packet is an IPv4 packet, accept it. */else if (ntohs(pktHdr->type) == ETHTYPE_IP) {TopLED(true); // debugTopLED(false);return true;}I have made the "multicast dropping" configurable via a config file entry, so it can be changed post-compilation, and curiously dropping non-ARP broadcast packets in low_level_input (as discussed above) doesnotavoid the need for more RX buffers at the PHY level. I still need 3+ there. So that issue is more subtle - assuming that my multicast dropping above is actually working. And I don't think it is since this code is never takenCode:// if the packet is to an IPv4 multicast destination, reject it.if (pktHdr->dest.addr == LL_IP4_MULTICAST_ADDR_0 &&pktHdr->dest.addr == LL_IP4_MULTICAST_ADDR_1 &&pktHdr->dest.addr == LL_IP4_MULTICAST_ADDR_2){TopLED(true); // debugTopLED(false);return false;}This bit of code does get taken every few seconds, which seems about rightCode:/* If the packet is a broadcast ARP accept it */if ( eth_addr_cmp(&pktHdr->dest, &ethbroadcast) ){if (ntohs(pktHdr->type) == ETHTYPE_ARP){TopLED(true); // todoTopLED(false);return true;}}I wondered why not a one-line conditional but then I realised that ntohs() is quite a lot of code so this is a good optimisation.This bit is not taken during normal running but is taken roughly once around the time DHCP etc is allocated, and it stops startup. I suspect there is some other ("pass") test missingCode:/* Reject all other packets */TopLED(true); //TopLED(false);return false; // this needs to be return true for things to work i.e. test bypassed}Another interesting data point is that low_level_input executes in 2us if there is no rx data, and 24-28us if there is a packet to be stuffed into LWIP. This puts a bit of perspective on whether an ISR is useful or not. 168MHz 32F417.

author: dare
author_rank: Contributor
timestamp: January 28, 2023, 12:49:08 pm
content: My apologies. I forgot you are initiating connections from your system. Try the following version of the function. It also includes logging, so you can see the decision being made along with the input data.Code:static bool should_accept_ethernet_packet(const uint8_t * pktData, size_t pktLen){const struct eth_hdr * pktHdr = (const struct eth_hdr *)pktData;bool accept;/* If the packet is too short, reject it immediately, without trying to log it. */if (pktLen < sizeof(struct eth_hdr)) {return false;}/* if the packet is to an IPv4 multicast destination, reject it. */else if (pktHdr->dest.addr == LL_IP4_MULTICAST_ADDR_0 &&pktHdr->dest.addr == LL_IP4_MULTICAST_ADDR_1 &&pktHdr->dest.addr == LL_IP4_MULTICAST_ADDR_2) {accept = false;}/* If the packet is a broadcast... */else if (eth_addr_cmp(&pktHdr->dest, &ethbroadcast)) {/* If the packet is an ARP packet, accept it. */if (ntohs(pktHdr->type) == ETHTYPE_ARP) {accept = true;}/* Reject all other broadcast packets. */else {accept = false;}}/* if the packet is a unicast ARP or IPv4 packet, accept it. */else if (ntohs(pktHdr->type) == ETHTYPE_ARP ||ntohs(pktHdr->type) == ETHTYPE_IP) {accept = true;}/* Reject all other unicast packets. */else {accept = false;}LWIP_DEBUGF(LWIP_DBG_ON | LWIP_DBG_TRACE,("should_accept_ethernet_packet: %s, dest:%"X8_F":%"X8_F":%"X8_F":%"X8_F":%"X8_F":%"X8_F", src:%"X8_F":%"X8_F":%"X8_F":%"X8_F":%"X8_F":%"X8_F", type:%04"X16_F"\n",(accept) ? "TRUE" : "FALSE",(unsigned char)pktHdr->dest.addr, (unsigned char)pktHdr->dest.addr, (unsigned char)pktHdr->dest.addr,(unsigned char)pktHdr->dest.addr, (unsigned char)pktHdr->dest.addr, (unsigned char)pktHdr->dest.addr,(unsigned char)pktHdr->src.addr, (unsigned char)pktHdr->src.addr, (unsigned char)pktHdr->src.addr,(unsigned char)pktHdr->src.addr, (unsigned char)pktHdr->src.addr, (unsigned char)pktHdr->src.addr,lwip_htons(pktHdr->type)));return accept;}The above code has been tested using the LwIP sample application running inside a linux process. I can confirm that it properly accepts DHCP, ARP, UDPv4 and TCPv4 traffic, including both initiating and accepting TCP connections.When looking at the log output, type:0800 indicates an IPv4 packet, type:0806 is an ARP packet and type:86DD is an IPv6 packet (which should be dropped). All FF's in the dest address indicates a broadcast packet.Quote from: peter-h on January 28, 2023, 09:33:31 amAs I mentioned earlier, this code protects the second queuing point for incoming packets; specifically, the input queue to the LwIP tcpip thread. It doesn't nothing to prevent the Ethernet controller from running out of receive buffers (which constitutes the first queuing point). To deal with that, you need to do one or more of the following: increase the polling frequency of your ethernetif_input task, switch to an interrupt-driven wake for the ethernetif_input task, and/or increase the number of receive buffers.QuoteGiven how short this is, I suspect the polling frequency is the dominate factor in whether the Ethernet controller runs out of buffers. If I understand correctly, the interval you've set is 10ms idle/2ms with traffic, which is fairly long.

author: peter-h
author_rank: Super Contributor
timestamp: January 28, 2023, 02:14:22 pm
content: That is great - all working.QuoteI am also doing NTP (client and server) and potentially other stuff. Do I need to put those in also? I thought most of that is done with UDP packets addressed to a specific IP, which was obtained previously somehow, possibly by DNS?QuoteI wonder why this is not a much more widespread issue.Those broadcast packets would trigger a pause which was never less than 1-3 seconds. This didn't look right as being the "first backoff timeout for a missing packet" for TCP, because that would be ridiculous and would really hit performance. I would expect the first backoff to be maybe 100-300ms and this not noticeable in most applications. And the length of the pause meant it was obviously generated within LWIP.QuoteYes. The problem is that with polling and using the osDelay() function (which is desirable because it releases control to run other RTOS tasks) is that a parameter of 1 gives you 0 to 1ms, so I use osDelay(2) if I actually want some sort of delay (1-2ms). But osDelay(1) will still release control. So I made the "fast poll" that value, and it all still seems to run fine. But it didn't help at all with the ETH buffer size at which LWIP starts to fall over (2 buffers), so I went back to (2) because it definitely provides more time for other RTOS tasks.Re interrupt driven RX, I don't want to put in that time right now because I am trying to get onto other stuff (this product is nearly done) but what would help would be a very simple ISR which just sets a "there is ETH RX data" flag. In my polling loop I could then execute that osDelay(2) only if that flag is not set. That would be a reasonable and safe compromise. Not sure how to set this up though. I have done timer ISRs before but never played with the ETH controller. The "HAL" code sets up an ISR which then tests every possible source, which is ridiculous. And the interrupt source needs to be cleared correctly so the ISR doing nothing may not work unless it just clears the IP bit or some such.OTOH, remaining in a delay-free polling loop until the ETH buffers are empty will do the same thing.

author: ejeffrey
author_rank: Super Contributor
timestamp: January 28, 2023, 03:54:15 pm
content: You might force the line rate down to 10 megabit. The network switch will have considerably more buffer capacity than your code so pushing some of the buffering back there might help.I'm not what the if freertos has this, but the way to use an ISR for this is to use something like a condition variable or event wait instead of sleep. Your network task would wait on the event or a timeout and the ISR would signal the event causing the wait to unblock and be scheduled according to its priority.

author: dare
author_rank: Contributor
timestamp: January 28, 2023, 04:53:41 pm
content: Quote from: peter-h on January 28, 2023, 02:14:22 pmBoth NTP and DNS are UDP, and typically unicast. So it shouldn't be a problem. However, if you intend to use mDNS, you will need to adjust the filter.QuoteI agree, this seems like something other than simple packet loss. I still think it would be helpful to see a debug log covering the entire pause period, including a few seconds of good communication on both sides of the pause.QuoteWell, in fact, this is exactly how the ST example code works. The interrupt fires and does little more that set a FreeRTOS semaphore. The ethernet input task (ethernetif_input) waits on this semaphore and calls low_level_input() in a loop when it fires. Not the most efficient design (it requires two context switches to deliver a packet from a dead stop). But it's much more responsive than polling.Have you looked at the code in the ST LwIP example? It seems to have all functionality needed to use interrupts.https://www.st.com/en/embedded-software/stsw-stm32070.htmlOne question for you: Are you aware of the issue with cache coherency and the DMA controller? Lot of discussion about it on the ST board (e.g.https://community.st.com/s/question/0D53W00001Z9K9TSAV/maintaining-cpu-data-cache-coherence-for-dma-buffers). Now maybe the code you're using has already fixes for this. But if it doesn't, then perhaps the problem is that sometimes the cache lines holding the DMA descriptors aren't being flushed to main memory promptly, meaning that the Ethernet DMA controller doesn't see them right away, causing a pause in reception ... just a thought.Seems like someone needs to put together an LwIP example app for the STM32F4xx with all the known issues fixed. There's one here for the STM32H7 here:https://community.st.com/s/question/0D50X0000C6eNNSSQ2/bug-fixes-stm32h7-ethernet. If I had a dev board with me I would take a run at writing this myself.

author: peter-h
author_rank: Super Contributor
timestamp: January 28, 2023, 08:06:41 pm
content: QuoteYes you can do that. I am not sure what the gain is though. If data arrives, youhaveto process it. So you may as well use an ISR to process it.QuoteI can get you that but it would be a huge amount of text.QuoteThat one is behind their usual "security" where they send you a link and you download it. I just have; it's a huge library. I will take a look at it later. There is a procedure for rolling this up into a Cube project, which I've never done. I've always just written new modules (.c and .h, mainly). But I will take a look.The trouble is that practically everything from ST doesn't actually work properly. A lot of the code is clearly written by people will little or no embedded experience. It works OK on the demo boards they sell. My "Monday afternoon guy" spend a year or two of Monday afternoons fixing the original ST code. I am glad this is now well behind and I have been able to get on with real stuff.Well, he has just spent a few months of Mondays on fixing MbedTLS...My problem is that I don't have a deep expertise of C, or so much other ARM32 stuff, so while I do good solid code in the end, it takes me a long time, and I have to pick my battles carefully.QuoteYes; this has come up lots of times but each time turned out to be irrelevant to the 32F4. Like the __DMB hereCode:if (bufcount == 1U){/* Set LAST and FIRST segment */heth->TxDesc->Status |=ETH_DMATXDESC_FS|ETH_DMATXDESC_LS;/* Set frame size */heth->TxDesc->ControlBufferSize = (FrameLength & ETH_DMATXDESC_TBS1);/* Set Own bit of the Tx descriptor Status: gives the buffer back to ETHERNET DMA *///__DMB();heth->TxDesc->Status |= ETH_DMATXDESC_OWN;/* Point to next descriptor */heth->TxDesc= (ETH_DMADescTypeDef *)(heth->TxDesc->Buffer2NextDescAddr);}I don't fully understand why. I think the 32F4 doesn't have the data cache. The only people who do arewekand a guy called "Piranha" on the ST forum who likes to tell people how stupid they areI think it is all M7 related. Quoting from that ST post:QuoteQuoteI've offered people bits of money on various occassionsI use freelancer.com quite a bit for little isolated projects. One really good guy wrote some client-side JS for file upload and download (for an HTTP server which I wrote myself, after a project which was based on ST's HTTPD was aborted due to the subcontractor running way past the budget) and for that it works well. Others wrote a USB VCP loopback tester (which I used extensively on this project; it has four serial ports and a 5th is a USB VCP), a graphical FreeRTOS stack space viewer, etc.I had a look at the interrupt code and found the ETH IRQ handler in the CT Cube code. There is only about one IRQ for the ETHCode:/**** @brief This function handles ETH interrupt request.* @param heth pointer to a ETH_HandleTypeDef structure that contains* the configuration information for ETHERNET module** This ISR is never called because all the stuff is polled. The vector points* to ETH_IRQHandler which currently doesn't exist.* In fact since nothing references this function, it gets optimised away.**/void HAL_ETH_IRQHandler(ETH_HandleTypeDef *heth){/* Frame received */if (__HAL_ETH_DMA_GET_FLAG(heth, ETH_DMA_FLAG_R)){/* Receive complete callback */HAL_ETH_RxCpltCallback(heth);/* Clear the Eth DMA Rx IT pending bits */__HAL_ETH_DMA_CLEAR_IT(heth, ETH_DMA_IT_R);/* Set HAL State to Ready */heth->State = HAL_ETH_STATE_READY;/* Process Unlocked */__HAL_UNLOCK(heth);}/* Frame transmitted */else if (__HAL_ETH_DMA_GET_FLAG(heth, ETH_DMA_FLAG_T)){/* Transfer complete callback */HAL_ETH_TxCpltCallback(heth);/* Clear the Eth DMA Tx IT pending bits */__HAL_ETH_DMA_CLEAR_IT(heth, ETH_DMA_IT_T);/* Set HAL State to Ready */heth->State = HAL_ETH_STATE_READY;/* Process Unlocked */__HAL_UNLOCK(heth);}/* Clear the interrupt flags */__HAL_ETH_DMA_CLEAR_IT(heth, ETH_DMA_IT_NIS);/* ETH DMA Error */if(__HAL_ETH_DMA_GET_FLAG(heth, ETH_DMA_FLAG_AIS)){/* Ethernet Error callback */HAL_ETH_ErrorCallback(heth);/* Clear the interrupt flags */__HAL_ETH_DMA_CLEAR_IT(heth, ETH_DMA_FLAG_AIS);/* Set HAL State to Ready */heth->State = HAL_ETH_STATE_READY;/* Process Unlocked */__HAL_UNLOCK(heth);}}and that then calls various other bits. I really don't want interrupt drivenTXbecause AIUI the packets just pass through LWIP (following a socket write call by the application) and end up in low_level_output, which immediately loads them into the ETH buffer(s). I spent some time measuring performance for different ETH TX buffer numbers and found anything above 2 is a waste of RAM (may not be for 4G etc).Thank you for the time you have spent helping me

author: Siwastaja
author_rank: Super Contributor
timestamp: January 29, 2023, 07:15:04 am
content: Also remember that TCP is supposed to be crappy over slow/unreliable links when one wants any kind of real-time responsiveness. Wrong protocol for the job if you can't accept a few seconds of delay after losing a packet; although you could make it better by tuning.Everyone who has tried to type command over SSH connection even with 1-2% packet loss or 500ms ping knows the pain.It has got better thanks to decades and decades of tuning, but tuning works given right protocol implementations and certain types of payloads. For example, I think SSH and HTTP work better now than 15-20 years ago; back then losing a few packets possibly meant waiting for so many seconds to the point of hitting F5 manually on a browser was faster than waiting for the TCP to recover!If you need quick response time under packet loss and/or small memory footprint, you are better off designing your own application-specific simple protocol. Usually on top of UDP. This is what games too.

author: peter-h
author_rank: Super Contributor
timestamp: January 29, 2023, 10:24:14 am
content: Yes; this is why initially I thought these long gaps are normal. And they rarely occur with short packets, presumably because short packets pass by faster. In the vast majority of applications would not be detected by the customer.But they should not be normal; for a start they are quite noticeable in any human user scenario. Or if running some serial to ethernet to serial application, you will need to set rather long timeouts to avoid errors.Note that data was not usually being lost. It was just being delivered with long gaps so e.g. a 1000 byte test packet would be delivered as 300 300 200 200 with gaps in between.Also AFAICT the timeout in my LWIP config is 250ms, not the "3 seconds" which tends to turn up on google when searching for a lost packet timeout. Also 3 secs would be quite ridiculous.I still think it is suspicious because the gaps are not even. But it is easy to reproduce. Just set buffers=2, recompile, and run, and right away you see it. It gets better after a minute or so, which makes sense as the frequency of broadcast packets reduces after an initial busy period when a new box is connected to the LAN.I am now back to running with the "busy" TP-Link QOS switch again, with all the other stuff on it including a VPN to another site, since it is a better test environment. That same switch produced a lot of trouble in the past:https://www.eevblog.com/forum/networking/bizzare-issue-with-a-linksys-lgs116p-qos-switch-fake-mac-related/msg3622359/#msg3622359My box is using a real MAC # this time.On a more bizzare topic: I've increased the ETHTXbuffers from 2 to 4 and it looks like I will have another interesting data point. But I won't know for sure until the evening. But surely this has to be pretty equivalent to givingLWIPmore TX buffers. But I can't see any lwipopts.h setting forTXbuffering (did many hours of googling in the past) hence I think there isn't one, and LWIP just takes the bufferyoupass to it when writing a socket and sends it either direct (if < MTU) or sends it in multiple chunks. It has to prefix and postfix each chunk on the way to the PHY layer, but AFAICT there is no "TX buffering" as such. Never found anybody who knew about this... but that is how I would do it. No point in buffering for fun, and this method also makes it possible to do "zero copy TX" which I think LWIP actually does. That also explains why a socket write is blocking for ever, even on a non blocking socket, until the next RTOS tick.Zero copyRXis a whole different animal, and has been done, but only with oblique references on forums and little if any working code.EDIT: the TX buffer increase benefit is hard to reproduce. I am not surprised because, during extensive previous tests, the data disappeared down the wire so fast that above 2 made no difference at all.

author: peter-h
author_rank: Super Contributor
timestamp: January 30, 2023, 01:27:43 pm
content: Had a quick look at the history of why this project doesn't use ETH RX interrupts.It was found that with the reduced (RMII) PHY interface, the LAN8742's IRQ output is shared with an LED driver and thus cannot be used to generate interrupts if you want the LED function. This may be nonsense because it does appear that the 32F4 can generate an IRQ wholly internally, regardless of whether the PHY is MII or RMII. But this was never tested. It may have been something to do with the ST development board originally used.If that IRQ works, it points to hereCode:void HAL_ETH_IRQHandler(ETH_HandleTypeDef *heth){/* Frame received */if (__HAL_ETH_DMA_GET_FLAG(heth, ETH_DMA_FLAG_R)){/* Receive complete callback */HAL_ETH_RxCpltCallback(heth);/* Clear the Eth DMA Rx IT pending bits */__HAL_ETH_DMA_CLEAR_IT(heth, ETH_DMA_IT_R);}/* Frame transmitted */else if (__HAL_ETH_DMA_GET_FLAG(heth, ETH_DMA_FLAG_T)){/* Transfer complete callback */HAL_ETH_TxCpltCallback(heth);/* Clear the Eth DMA Tx IT pending bits */__HAL_ETH_DMA_CLEAR_IT(heth, ETH_DMA_IT_T);}/* Clear the interrupt flags */__HAL_ETH_DMA_CLEAR_IT(heth, ETH_DMA_IT_NIS);/* ETH DMA Error */if(__HAL_ETH_DMA_GET_FLAG(heth, ETH_DMA_FLAG_AIS)){/* Ethernet Error callback */HAL_ETH_ErrorCallback(heth);/* Clear the interrupt flags */__HAL_ETH_DMA_CLEAR_IT(heth, ETH_DMA_FLAG_AIS);}}(I've removed the superfluous code)Looking at the interrupt config in the RM, there is a vast number of interrupts which can be enabled, and a vast range of IP bits which need to be cleared accordingly. So unless somebody really understands the hardware, there is a lot of code that needs to be done right.

author: Daixiwen
author_rank: Frequent Contributor
timestamp: January 31, 2023, 08:36:18 am
content: I'm not sure I understood what you wrote correctly, but the interrupt from the LAN8742 is only used for events from the PHY itself. Those are usually status changes, such as autonegotiation result and cable connect/disconnect. You can see the interrupt sources in the datasheethereon page 87. The Ethernet receive interrupt is completely different and comes from the MAC or the DMA, and as you describe is coming from the 32F4 itself. It should have nothing to do with the PHY IRQ line.

author: peter-h
author_rank: Super Contributor
timestamp: January 31, 2023, 08:48:23 am
content: Thanks - that makes sense. So it was a misunderstanding right at the start. Currently we poll the status, by calling this about once a secondCode:// Called from ETHIF thread. Detects link change and resets the link state.static void ethernetif_set_link(struct netif *netif){uint32_t regvalue = 0;// Read PHY_BSRHAL_ETH_ReadPHYRegister(&EthHandle, PHY_BSR, &regvalue);regvalue &= PHY_LINKED_STATUS;// Check whether the netif link down and the PHY link is upif (!netif_is_link_up(netif) && (regvalue)){/* network cable is connected */netif_set_link_up(netif);}else if (netif_is_link_up(netif) && (!regvalue)){/* network cable is disconnected */netif_set_link_down(netif);}}IOW, this reads the LAN8742 status reg and sees if it has changed.This is all done in low_level_input which as I mention further back is polled every 10ms, reducing to 2ms if there is rx data, reverting to 10ms after 10ms of no data.

author: Daixiwen
author_rank: Frequent Contributor
timestamp: January 31, 2023, 09:06:49 am
content: Yes indeed once a second is more than enough. How quickly do you want your application to react to a cable connect / disconnect?I am not familiar with the 32F4 (or any STM32 in face), but there should be a "new packet" interrupt from the MAC or the RX DMA that you could use instead of polling to have a better performance. From your code snippet it could be ETH_DMA_IT_R but it would need some testing to see how it needs to be setup.

author: peter-h
author_rank: Super Contributor
timestamp: January 31, 2023, 09:55:38 am
content: Indeed; as discussed further back, interrupt RX would deliver a higher performance. I just don't want to do it right now because I don't think there is any proven working code available (for the 32F4; various code snippets have appeared for others) and the ST Cube MX demo code (for their demo board(s)) worked but was not reliable.There will always be the issue that with a broadcast storm at 100mbps you get interrupts at - depending on actual packet size - around 500kHz (2us), but the time to extract a packet from the ETH PHY buffers has been measured at way more than that. Genuine packets take ~30us to process (from ETH PHY buffers to the back end of LWIP, using memcpy in 32 bit chunks; DMA would not help especially on small packets) and non-ARP broadcasts, while discarded (unless this is configured OFF, which it may be if you want to do "UPNP" i.e. product discovery) take about 10us. Even the simplest code to service an interrupt and discard the packet based on its header will be ~10us including the ISR overhead. So it would be quite easy to totally kill (DOS) this product on a bad LAN, unless more complication is introduced.Link change detect at say 1Hz is easily ok.

author: Daixiwen
author_rank: Frequent Contributor
timestamp: January 31, 2023, 11:58:24 am
content: Yes I can understand the problem.Another easier optimization that I have used, since the driver seems to be using several receive buffers, is to make sure that the DMA already reads the next packet while you are processing the current packet. Then you have the possibility to poll the rx status right after you are finished processing one packet, and under heavy traffic it's probable you would already have a new packet waiting for you. The main problem with this solution, a bit like what you mentioned with the interrupts, is that you may make the application vulnerable to DOS by sending a lot of packets, as the receive task usually has a higher priority than the rest of the application. You could have a counter so that the receive task yeilds control back to the OS after n packets and find the right balance between performance and vulnerability.

author: peter-h
author_rank: Super Contributor
timestamp: January 31, 2023, 12:26:13 pm
content: Yes; I have tested this too. In the polled model, it is easy. This is the entire RTOS task for ETH RX:Code:void ethernetif_input( void * argument ){struct pbuf *p;struct netif *netif = (struct netif *) argument;uint32_t link_change_check_count = ETH_LINK_CHANGE_COUNT;// Define RX activity timer, for dropping fast poll down to slow pollTimerHandle_t *rxactive_timer = xTimerCreate("ETH RX active timer", pdMS_TO_TICKS(ETH_SLOW_POLL_DELAY), pdFALSE, NULL, RXactiveTimerCallback);// Start "rx active" timerxTimerStart(rxactive_timer, 20);// 20 is just a wait time for timer allocationdo{p = low_level_input( netif );// This sets rxactive=true if it sees dataif (p!=NULL){if (netif->input( p, netif) != ERR_OK ){pbuf_free(p);}}if (rxactive){rxactive=false;// Seen rx data - reload timeoutxTimerReset(rxactive_timer, 20);// Reload "rx active" timeout (with ETH_SLOW_POLL_DELAY)// and get osDelay below to run fastrx_poll_period=ETH_RX_FAST_POLL_INTERVAL;}// This has a dramatic effect on ETH speed, both ways (TCP/IP acks all packets)osDelay(rx_poll_period); // 2ms or 10ms// Do ETH link status change checklink_change_check_count--;if (link_change_check_count==0){// reload counterlink_change_check_count = ETH_LINK_CHANGE_COUNT;// Get most recently recorded link statusbool net_up = netif_is_link_up(&g_netconf_netif);// Read the physical link statusethernetif_set_link(&g_netconf_netif);// Has the link status changedif (net_up != netif_is_link_up(&g_netconf_netif)){ethernetif_update_config(&g_netconf_netif);if (net_up) {// Link was up so must have droppeddebug_thread_printf("Ethernet link down");}else {// Link was down so must be up - restart DHCPdebug_thread("Ethernet link up");network_restart_DHCP();}}}} while(true);}Half of it is the link change detect stuff, so forget that.It is easy to rig this so that if p!=NULL hereCode:if (p!=NULL){if (netif->input( p, netif) != ERR_OK ){pbuf_free(p);}}the call below to osDelay() is skipped, so you go right back to testing if there is another packet (and probably skip the link change stuff as well).That that takes you right back to being easily DOSable. So what can one do? As you suggest, a counter for the max # of consecutive packets thus retrieved? That will give good performance on a burst of packets. But only if the burst is "just right".Much depends on what other code you are running, because obviously all the time you are in this loop you won't be yielding back to the RTOS (with osDelay(1) being the simplest example) so at 1kHz Systick nothing else will run for 1ms, except ISRs.You could spend your whole life optimising some scenariobut because this CPU doesn't have a hardware PHY-layer packet filter, it will never work properly is you get a packet flood. The 32F4 can dump broadcast packets but that's OK only (AFAICT) for a box which is addressed by the MAC #.

author: nctnico
author_rank: Super Contributor
timestamp: January 31, 2023, 12:31:41 pm
content: Yep. On a constrained platform you have to just drop ethernet packets at some point. Every protocol that runs on ethernet is designed to deal with packet loss so you won't lose any functionality. Only realtime performance will suffer. In some of the ethernet based designs I have made, the MAC interface is bitbanged using GPIO pins (with a switch chip as PHY that can also do buffering).

author: peter-h
author_rank: Super Contributor
timestamp: January 31, 2023, 01:21:50 pm
content: QuoteExcept LWIP which seemingly can't handle a broadcast packet at the same time as some other packet... Perhaps this issue occurs only if- you have just 2 buffers- 1 gets a broadcast- 1 gets some other- and then there is a gapOtherwise, in my test setup, with broadcasts at ~1Hz, this would be happening at the broadcast frequency, but actually it is very rare.Or maybe it is a bug in some of the low level code, which disappears when you have 3+ buffers.QuoteIsn't that very slow?

author: nctnico
author_rank: Super Contributor
timestamp: January 31, 2023, 01:43:06 pm
content: Quote from: peter-h on January 31, 2023, 01:21:50 pmFaster compared to a 14k4 modem

author: peter-h
author_rank: Super Contributor
timestamp: January 31, 2023, 03:27:43 pm
content: Sure, TCP/IP delivers end to end error correction and flow control, which makes this 2-buffer+broadcast thing even more of a mystery.At the PHY level, a broadcast is handled like any other packet, AIUI. Only inside LWIP is there any difference. Is that right?Several days of testing now, with no errors.

author: ejeffrey
author_rank: Super Contributor
timestamp: January 31, 2023, 08:50:21 pm
content: This is another area where a packet sniffer would be extremely helpful. You could see exactly what broadcast (or other!) packets are being sent closely with the data that is being dropped. It might be that some part of the network is e.g., triggering an ARP request synchronous with your data packets, causing multiple sequential overflows even on a relatively low utilization link.TCP interprets packet loss as congestion, and uses an exponential backoff algorithm to avoid making the congestion worse by sending lots of retries. That assumption breaks down when the packet drops are not really due to congestion. This comes up with wireless communication where interference or signal fade can cause a burst of packet drops, which can cause all the TCP sessions to back off. Once the link comes back, it may be immediately available at 100% capacity, but it takes a while for all the stalled TCP connections to ramp back up.2 packet buffers gives you only a small margin for error, and also your 10 ms polling interval could also be causing problems. In some sense it's not surprising that you are having trouble, but it's also hard to say without seeing exactly what is happening.

author: peter-h
author_rank: Super Contributor
timestamp: January 31, 2023, 10:27:15 pm
content: FWIW, I tried a near-zero polling interval and it did the same thing.Also I am running a test with 2 sessions, with randomly timed packets, and for sure there would be 2 packets arriving "concurrently" and grabbing those two buffers, quite often.Yet the problem surfaced only with broadcasts present.The problem is that LWIP is immensely complicated. The only thing more horrible is MbedTLS, although to be fair 90% of that is crypto suites which they pulled in from all over the net. Half of the config in lwipopts.h (and opt.h) is a mystery, evidently to a zillion devs desperately posting all over the www over past 15+ years. I guess if you know tcp/ip from the fundamentals (like a 1970s Computer Science academic) then you will recognise most of it because your brain is already "wired right". FWIW I recompiled the project with every single LWIP option doubled up, all tested separately, and nothing made any difference. This thing is something deep down. May have been some task deadlock between LWIP and the RTOS, and if it was, it would never be fixed; not by somebody on my pay grade.Anyway it is fixed now

author: Daixiwen
author_rank: Frequent Contributor
timestamp: February 01, 2023, 09:45:21 am
content: Yes it's hard to follow the code in LWIP. I try to avoid it as much as I can. But this behavior looks like a bug.Maybe you could implement your own filter in the eth rx function, that would automatically drop any broadcast packet that isn't an ARP request, before giving them to LWIP? It feels a bit like fixing a problem with rope and gaffer tape, but sometimes you just need to move forward with the project...

author: peter-h
author_rank: Super Contributor
timestamp: February 01, 2023, 10:10:08 am
content: Yes that was done - see posts bydarefurther back.I have this in there, user configurable via a config file entry.That filterdoesenable reliable operation with eth rx buffers=2 but not completely so, so even ARP broadcasts cause the problem (though far less often).So it isn't a CPU issue and in any case I don't think the CPU treats broadcasts differently (if that feature to dump them is not used).Probably some deadlock issue with lwip+rtos. For example, when playing around, I found that you canreaddata from a socket with rtos task switching disabled, but if you try a socketwrite, it hangs. So a non blocking socket does actually block until the next rtos tick. I can see why; lwip needs to package the data and send it out, which it can't do unless there is room lower down, and any kind of state machine runs on ticks.

author: Daixiwen
author_rank: Frequent Contributor
timestamp: February 02, 2023, 08:54:38 am
content: Sorry about that. I have to admit I only read the beginning of the thread a while back and the messages from last week. I just had a glance over the rest and missed it.I have more experience with the BSD IP stack and the eCos operating system, but could your problem be related to priority inversion? We had the problem once on an application because the Ethernet RX thread was running on the highest priority, the rest of the IP stack on a high priority and the application on a lower priority. Without priority inheritance we had all kind of weird problems when the network traffic got higher, especially when trying to send and receiving at the same time. Ideally the RX and TX paths should use mutually exclusive synchronization mechanisms but it's not possible with every kind of hardware out there.If you have at least 3 threads on different priorities all using the same mutex/semaphore for synchronization it may be something to look into:https://www.digikey.no/en/maker/projects/introduction-to-rtos-solution-to-part-11-priority-inversion/abf4b8f7cd4a4c70bece35678d178321

author: peter-h
author_rank: Super Contributor
timestamp: February 02, 2023, 09:54:11 am
content: Yes; could be.The lowest level ETH RX (LOW_LEVEL_INPUT) runs at osPriorityRealtime = 48. This stuffs the eth rx packets into LWIP. No mutexes are involved with this thread.LWIP runs at osPriorityHigh = 40.Applications are supposed to run at a lower priority than osPriorityHigh and mine is running at tskIDLE_PRIORITY = 0.The question is whether a lower priority task can set a mutex which blocks a higher priority task. The only mutex I can think of which might be active here is the LWIP one, set up in lwipopts.h:Code:// Flag to make LWIP API thread-safe. The netconn and socket APIs are claimed// to be thread-safe anyway. The raw API is never thread-safe.#define LWIP_TCPIP_CORE_LOCKING1This is poorly documented but tracing the code down, it appears to set up a mutex around the API calls. This implements quite a coarse exclusion than what LWIP is claimed to do without this mode set but is claimed to be more efficient because LWIP implements thread-safety by using a messaging system internally which takes up more processing. That's my primitive understanding; I never found anyone who could give a straight answer (despite vast numbers of questions on the www, and on the nearly-dead LWIP mailing list). Anyway, using this option or not makes no difference to the issues I had.But there is onlyonemutex for LWIP and I can see that if you lock a socket on e.g. a read then another thread doing any socket operation will get blocked. This will obviously be a problem if someone is using ablockingread socket; these hang until data arrives. For writes, this should not hang anything because AIUI all socket writes end up with data disappearing down the wire anyway.This description is curious - here:https://www.nongnu.org/lwip/2_0_x/group__lwip__opts__lock.htmlLWIP_TCPIP_CORE_LOCKING Creates a global mutex that is held during TCPIP thread operations. Can be locked by client code to perform lwIP operations without changing into TCPIP thread using callbacks. See LOCK_TCPIP_CORE() and UNLOCK_TCPIP_CORE(). Your system should provide mutexes supporting priority inversion to use this.Does anyone know what "without changing into TCPIP thread using callbacks" means?I will continue testing with LWIP_TCPIP_CORE_LOCKING = 0 and see if anything breaksCurrently I have several RTOS tasks all using either the Netconn API (an HTTP server I wrote) or the socket API, all running fine.

author: peter-h
author_rank: Super Contributor
timestamp: February 04, 2023, 09:32:02 am
content: Just an update: further input on LWIP_TCPIP_CORE_LOCKING suggests 0 (disabled) should not have any advantages in any circumstances compared with 1 (enabled).I thought that testing showed better reliability under heavy load (multiple RTOS tasks all using netconn and sockets) with 0 but can't really replicate it, and it could be mixed up with PBUF buffer allocation dependencies.1 uses a mutex at a high level - at the API level - so will lock out TCP tasks at that high level. This may not matter, but could result in "priority inversion" in that a low priority task using TCP will block a higher priority one, for the entire duration of the LWIP API call. I now possibly understand the bit herehttps://www.nongnu.org/lwip/2_0_x/group__lwip__opts__lock.htmlYour system should provide mutexes supporting priority inversion to use this.I have no idea what such a mutex would be (a mutex is a mutex, and has to be used with care, no?) but whether this matters depends. It doesn't make sense for a task using TCP comms to have a very high priority anywayAnd for sure the effect will depend on how much buffering you have allocated to the ETH PHY TX; I don't think LWIP has anyTXbuffering configured anywhere. Helping this will be that most if not all packets disappear down the wire at 10mbytes/sec regardless, especially if Nagle is disabled which seems a good idea in embedded apps anyway.Dare's multicast filtering function works wonderfullyThe only possible issue is that if one day I want to implement discovery, then processing broadcasts will be necessary. Then I guess the function would be modified with accept broadcast packets with a specific payload.Later editin case somebody finds this in the future:The question of whether LWIP is thread safe is nontrivial. Apparently it is even without LWIP_TCPIP_CORE_LOCKING=1, for the netconn and socket APIs but it is never thread-safe for the raw API. Anyway, originally the extra locking has been implemented as it is claimed to improve performance. More info:https://www.nongnu.org/lwip/2_1_x/multithreading.htmlhttps://www.nongnu.org/lwip/2_1_x/pitfalls.htmlThe 2nd URL above says the RAW API is also thread-safe if core locking=1.While the consensus is that LWIP_TCPIP_CORE_LOCKING=1 should be the default usage, it was experimentally found that the system is much more reliable with =0, so these two have been set to 1 as per the above docs.#define LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT 1#define SYS_LIGHTWEIGHT_PROT 1This is necessary especially as, in the commonly used ex-ST low_level_input and low_level_output code, other RTOS tasks are calling e.g. pbuf_free() in ethernetif.c.Why the system is much more reliably with LWIP_TCPIP_CORE_LOCKING=0 I don't know. The difference is found only when I have around 5 RTOS tasks running, some HTTP, some HTTPS, and other stuff, some using netconn API and some using the socket API.

author: dare
author_rank: Contributor
timestamp: February 04, 2023, 09:56:29 pm
content: Quote from: peter-h on February 04, 2023, 09:32:02 amTo avoid any potential confusion, readers should know that this statement is not true, per se. That is, the raw (a.k.a. "low-level") APIs arenotinherently thread-safe in that their implementations do not contain code to handle contention between threads. However, when the LwIP core locking feature is enabled (LWIP_TCPIP_CORE_LOCKING = 1), these APIs may be called from a thread other than the LwIP "tcpip" thread by acquiring the global core lock before calling the function. When used this way, it is entirely up to the caller to ensure that the lock is acquired and released at the correct times. (It is also important to know that the raw API is callback-based, and that all callbacks that occur will happen on the tcpip thread, regardless of which the thread made the original API call).Finally, a note about terminology from the LwIP docs:Do not confuse the lwIP raw API with raw Ethernet or IP sockets.The former is a way of interfacing the lwIP network stack (includingTCP and UDP), the later refers to processing raw Ethernet or IP datainstead of TCP connections or UDP packets.

author: peter-h
author_rank: Super Contributor
timestamp: February 04, 2023, 11:02:33 pm
content: ThanksI was reading this up today and especially in light of calling pbuf_free() from an RTOS task which isnotthe TCP/IP task, hereCode:do{p = low_level_input( netif );// This sets rxactive=true if it sees dataif (p!=NULL){if (netif->input( p, netif) != ERR_OK ){pbuf_free(p);}}if (rxactive){rxactive=false;// Seen rx data - reload timeoutxTimerReset(rxactive_timer, 20);// Reload "rx active" timeout (with ETH_SLOW_POLL_DELAY)// and get osDelay below to run fastrx_poll_period=ETH_RX_FAST_POLL_INTERVAL;}// This has a dramatic effect on ETH speed, both ways (TCP/IP acks all packets)osDelay(rx_poll_period);// Do ETH link status change check (code not shown for brevity)} while(true);Now testing withCode:// Flag to make LWIP API thread-safe. The netconn and socket APIs are claimed// to be thread-safe anyway. The raw API is never thread-safe.// A huge amount of online discussion on this topic; most of it unclear, but// ON (1) seems to be recommended, as being more efficient. However, it was// found that OFF (0) greatly reduces weird data loss errors under heavy// multiple TCP/IP RTOS task load.#define LWIP_TCPIP_CORE_LOCKING0// If LWIP_TCPIP_CORE_LOCKING=0 then these two need to be 1// See https://www.nongnu.org/lwip/2_1_x/multithreading.html#define LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT 1#define SYS_LIGHTWEIGHT_PROT 1Looking at the actual code on e.g. a socket write, when LWIP_TCPIP_CORE_LOCKING=1, the whole API call is locked and the granularity is pretty poor, which should still work but probably doesn't help...

author: peter-h
author_rank: Super Contributor
timestamp: February 08, 2023, 09:33:30 am
content: Just had loads of fun and made some interesting progress.There were two reasons for the multi second gaps:- broadcast packets (2 buffers, one with a bcast and one with data, messes up LWIP) - fundamental reason not established but bcasts are now filtered, and there are 4 rx buffers- RTOS task priorityOn the 2nd one:Notes from my ref manual:Most projects can be written with all "application" tasks running at tskIDLE_PRIORITY, yielding to RTOS when waiting for something.One exception to the above is where you have higher priority task(s) running which are not yielding, and you do not want execution gaps in the lower priority tasks. In the xxx this happens if TLS is used. TLS is a complex monolithic task which performs crypto operations lasting up to several seconds. It runs in the priority of whichever task called TLS, and typically this is network_task (LWIP itself, priority osPriorityNormal = 24). If these gaps would be an issue,your task priority needs to be same or higher than network_task(24). Where TLS is not used, the task priority can be all the way down to tskIDLE_PRIORITY, even where networking is used.Furthermore, due to the structure of LWIP, networkin application priorityshould not be higherthan LWIP (24) since it might cause blocking on socket writes.The above is because TLS is being called from the LWIP task, basically. Somebody else may have done it differently... maybe, with inter-task message passing, but why? I suppose there is an argument for running TLS at a very low priority, if you are happy with gaps in the comms / slow comms, but don't want TLS causing the multi second gaps.The other approach is to rehack TLS so it yields periodically. Then you have a whole loooad of fun rehacking TLS every time you want to update itAnd ours has already been hacked around to process a string of certificates from a file without needing > 200k of RAM to hold the whole current cert chain.

author: peter-h
author_rank: Super Contributor
timestamp: February 09, 2023, 09:15:36 pm
content: And here is another puzzle. Two sections of the ETH - LWIP low_level_* code. Each one contains two memcpy calls. In each case, the 1st call of the two is never taken...low_level_input:Code:// Load the packet (if not rejected above) into LWIP's bufferif (p != NULL){dmarxdesc = EthHandle.RxFrameInfos.FSRxDesc;bufferoffset = 0;for(q = p; q != NULL; q = q->next){byteslefttocopy = q->len;payloadoffset = 0;/* Check if the length of bytes to copy in current pbuf is bigger than Rx buffer size */while( (byteslefttocopy + bufferoffset) > ETH_RX_BUF_SIZE ){/* Copy data to pbuf */memcpy( (uint8_t*)((uint8_t*)q->payload + payloadoffset), (uint8_t*)((uint8_t*)buffer + bufferoffset), (ETH_RX_BUF_SIZE - bufferoffset));/* Point to next descriptor */dmarxdesc = (ETH_DMADescTypeDef *)(dmarxdesc->Buffer2NextDescAddr);buffer = (uint8_t *)(dmarxdesc->Buffer1Addr);byteslefttocopy = byteslefttocopy - (ETH_RX_BUF_SIZE - bufferoffset);payloadoffset = payloadoffset + (ETH_RX_BUF_SIZE - bufferoffset);bufferoffset = 0;}/* Copy remaining data in pbuf */memcpy( (uint8_t*)((uint8_t*)q->payload + payloadoffset), (uint8_t*)((uint8_t*)buffer + bufferoffset), byteslefttocopy);bufferoffset = bufferoffset + byteslefttocopy;}}low_level_output:Code:/* Check if the length of data to copy is bigger than Tx buffer size*/while( (byteslefttocopy + bufferoffset) > ETH_TX_BUF_SIZE ){// Copy data to Tx buffer - should use DMA but actually the perf diff is negligiblememcpy_fast( (uint8_t*)((uint8_t*)buffer + bufferoffset), (uint8_t*)((uint8_t*)q->payload + payloadoffset), (ETH_TX_BUF_SIZE - bufferoffset) );/* Point to next descriptor */DmaTxDesc = (ETH_DMADescTypeDef *)(DmaTxDesc->Buffer2NextDescAddr);/* Check if the buffer is available */if((DmaTxDesc->Status & ETH_DMATXDESC_OWN) != (uint32_t)RESET){errval = ERR_USE;goto error;}buffer = (uint8_t *)(DmaTxDesc->Buffer1Addr);byteslefttocopy = byteslefttocopy - (ETH_TX_BUF_SIZE - bufferoffset);payloadoffset = payloadoffset + (ETH_TX_BUF_SIZE - bufferoffset);framelength = framelength + (ETH_TX_BUF_SIZE - bufferoffset);bufferoffset = 0;}/* Copy the remaining bytes */memcpy( (uint8_t*)((uint8_t*)buffer + bufferoffset), (uint8_t*)((uint8_t*)q->payload + payloadoffset), byteslefttocopy );bufferoffset = bufferoffset + byteslefttocopy;framelength = framelength + byteslefttocopy;}

author: dare
author_rank: Contributor
timestamp: February 09, 2023, 11:48:49 pm
content: Quote from: peter-h on February 09, 2023, 09:15:36 pmThis is because the size of packets you are sending/receiving are smaller than the size of the MAC buffers you are using to hold them (as specified by ETH_TX_BUF_SIZE and ETH_RX_BUF_SIZE values).Indeed, based on what you posted earlier, this will always be the case, since ETH_TX_BUF_SIZE and ETH_RX_BUF_SIZE are both set to ETH_MAX_PACKET_SIZE. So the while loop in both functions is unnecessary.Relatedly, the for loop in low_level_input() is also unnecessary. This is because the size of the buffers in the pbuf pool (as set by PBUF_POOL_BUFSIZE) is larger than the maximum Ethernet frame that can be received. Therefore, when pbuf_alloc() is called to allocate a buffer of the desired size (len), it will only ever return a single buffer (i.e. a buffer where buf->next == NULL).

author: dare
author_rank: Contributor
timestamp: February 10, 2023, 12:57:11 am
content: Quote from: peter-h on February 08, 2023, 09:33:30 amIf this is true, it means you're running mbedTLS in an extremely inefficient mode. Typically, when using mbedTLS with LwIP in an RTOS context (i.e. NO_SYS==0), mbedTLS runs in the application task. In this configuration, mbsdTLS calls the LwIP sockets API to send and receive TLS "records" (a.k.a. messages) over the underlying transport connection. In this way, all TLS computations, including crypto operations, are performed in the application task, and the lwip tcpip task is solely responsible for attending to the TCP state (e.g. sending/receiving ACKs, performing retransmissions, etc.).There is a configuration that allows the use of mbedTLS directly on top of the LwIP raw API. This mode uses the altcp_tls API. Because its built on the raw API, the atltcp_tls API can only called from the tcpip task. Unfortunately, this configuration is really only suitable for very simplistic applications, and even then it is sub-optimal.As you've observed, the fundamental problem is that the crypto math takes a long time. This is especially true on the STM32F4s which have no acceleration for RSA or EC operations. What this means is that, while the tcpip task is performing a crypto operation it is unable to perform any of the other operations for which it is responsible. As a consequence, if you have other application tasks which are using the sockets API, these tasks will be blocked waiting for the tcpip task to finish the crypto op and attend to the socket request. Even in a system where there are no other application tasks, and all network activity happens in the tcpip task, running the crypto code in the tcpip task itself is a bad idea, because means that other normal TCP activities like sending delayed ACKs or retransmissions will not happen in a timely manner, leading to poor throughput. Yielding from inside the TLS code wont help with this problem, since the tcpip thread is trapped inside the crypto code and can't get back to its main loop to service other activities.

author: peter-h
author_rank: Super Contributor
timestamp: February 10, 2023, 07:31:43 am
content: QuoteI tried to create large packets (I am testing with various programs running e.g. an http server doing file downloads and uploads) and nothing triggers that condition, which makes me wonder if the conditionwhile( (byteslefttocopy + bufferoffset) > ETH_TX_BUF_SIZE )caneverbe met. Is there any point in having ETH_TX_BUF_SIZE smaller, anyway? It would impose a limit on the MTU, for a start.I struggle to see where the size of the buffer is defined. For RX, it should be the (already negotiated) MTU, and ETH_RX_BUF_SIZE is already equal to the largest possible "standard" MTU. For TX, it is whatever LWIP sends to ETH which is probably going to be again the MTU configured in lwipopts.h.So that code will get run only if somebody is running with small ETH packets.I am using a large PBUF_POOL_BUFSIZE (1514) but still just under ETH_RX_BUF_SIZE.Relative to thishttps://www.eevblog.com/forum/microcontrollers/gcc-compiler-optimisation/msg4692329/#msg4692329I am wondering if there is something I can do to improve the chance of 4-aligned memcpy transfers. The ETH_ buffers are aligned but I am not sure about the LWIP ones. Obviously a sequence of 1514 byte buffers won't be. I went to 1516.The payloadoffset is not usually going to be 4-aligned (not aligned 75% of the time) but the buffer itself could be.QuoteEven more so with 1516Amazing. But I will leave the code in there...An original version of the Cube code had PBUF_POOL_SIZE=8 and PBUF_POOL_BUFSIZE=512. I spent a lot of time playing with this and ETH performance and found the 8x512 to have no advantage. AIUI LWIP would just concatenate 3 of those for large packets. But even running with this setting for a bit, I found that the 1st memcpy is still never taken. Could be a timing thing.There is a weird reason for using large pbufs: my simplistic http server is assuming that the entire "string of interest" (the GET... etc) is contained in one packet from LWIP. This works fine on a LAN, which is all that that server is for (local config). I need PBUF_POOL_BUFSIZE > ~500 to receive this kind of stuff in one goCode:* PUT /ufile=BOOT.TXT HTTP/1.1..Host: 192.168.3.73:81..User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64;* x64; rv:103.0) Gecko/20100101 Firefox/103.0..Accept: .....Accept-Language: en-US,en;q=0.5..Accept-En* coding: gzip, deflate..Content-Type: text/plain;charset=UTF-8..Content-Length: 198..Origin: http * p* ://192.168.3.73:81..Connection: keep-alive..Referer: http://192.168.3.73:81/efile?BOOT.TXT....boot* time: 2022-08-02 15:39:28.app name: appname_1.1** The likelihood of the 1st packet containing the entire header including the CRLFCRLF* is dependent on the value PBUF_POOL_BUFSIZE on lwipopts.h, and on the program sending* the data (the browser).*QuoteI think I was wrong. The application calls the MbedTLS API which in turn calls LWIP.QuoteWhat I observe by waggling GPIO pins and watching on a scope which tasks are running is that a task whose priority is same (or higher - I am not sure that is a good idea for other reasons, so I am staying with "same") as LWIP's doesnotget blocked by the RSA/EC delayeven if that task is using LWIP. But those tasks are merely using LWIP,notusing TLS.The tasks I have (2 of them) whichareusing TLS do get blocked and must obviously get blocked because RSA/EC is the session setup, before data can be sent. No way around that (other than a coprocessor).The problem with blocking a non-TLS "internet task" for 3 secs is that it makes things time out all over the place. This appears to be avoided by matching its priority to LWIP's.However this doesn't appear to work for LWIP applications which use the Netconn API. They get hung up. Socket API applications seem to run OK through the gaps produced by TLS crypto. That seems strange because the socket API uses the netconn API; this is clear when one steps through it.

author: dare
author_rank: Contributor
timestamp: February 10, 2023, 05:36:52 pm
content: Quote from: peter-h on February 10, 2023, 07:31:43 amIn your case I believe this expression will never be true. Because LwIP has been configured with an Ethernet MTU of 1500 (payload) bytes, the largest frame that LwIP will attempt to send (and therefore the largest value for byteslefttocopy) is 1518 bytes. The ST ETH driver sets ETH_TX_BUF_SIZE to ETH_MAX_PACKET_SIZE, which in turn is set to 1524 bytes (see stm32f4x7_eth.h). (Technically, the Ethernet max frame size is 1522; the driver code includes 2 "extra" bytes for reasons I haven't investigated). Thus the expression will never be true.QuoteHaving a smaller buffer size would not limit the MTU, because the MAC allows Ethernet frames to be span multiple buffers. Indeed, that's what the while loop is for--handling the case where the frame needs to be copied into more than one TX buffer. The benefit of smaller TX buffers is potentially more efficient memory usage in some cases. Many TX packets will be smaller than the Ethernet MTU (e.g. TCP ACKs). Having your total TX buffer space split up into smaller individual buffers allows more packets to be on the TX queue simultaneously. This might allow you to get away with less overall TX buffer memory in the case where you have many simultaneous TCP connections all vying for a spot on the TX queue. That said, if one is really concerned about reducing overall buffer memory size, the optimal answer is to do DMA directly into/out of the LwIP buffers themselves.QuoteAs you state, for inbound packets the max size will be the standard max Ethernet frame size (1522), possibly reduced by the negotiated TCP MSS. For outbound packets, the max size will be limited by the smaller of: the mtu value set on the netif, the value set for TCP_MSS in lwipopts.h (for TCP only), and the negotiated TCP MSS (again, for TCP only).QuoteAll the buffers in the LwIP pbuf pool are automatically aligned to whatever MEM_ALIGNMENT is set to. (This is thanks to the convoluted preprocessor logic in memp.h, memp_std.h and memp.c). This means on the RX side the memory copy from the ETH RX buffer to the LwIP pbuf will always be aligned.The TX side is more complicated. Depending on how your application sends its data, the outbound Ethernet frame as seen by low_level_output() may in fact be split up over multiple pbufs (thus requiring multiple iterations of the for loop to copy). This should only happen in cases where the application chooses the no-copy option when sending data. In those cases, one of the pbufs in the chain will point directly to the application's data buffer. Thus the alignment of this data will depend on whether the application's buffer was aligned to begin with.In most cases the data in pbufs will be aligned. So its best to use a copy function that is optimal for aligned data, but can cope with copying unaligned data.QuoteBe careful, because this depends on how the sending systemchoosesto write the HTTP header data. There's nothing in the HTTP spec that requires the sender to transmit the head of an HTTP request in a single TCP segment, even if the TCP MSS is big enough to support this. The only safe approach is to accumulate the received data into a buffer until the end-of-head marker (\r\n\r\n) is seen.QuoteFor good comms performance, typically you want your tcpip task to be higher priority than your application tasks, but lower priority than your timing-critical tasks (e.g. a task delivering audio to a DAC). The tcpip task should never block on anything other than its input work queue (and possibly the tcpip global lock, if core locking is enabled). The only CPU intensive work in the tcpip task is data copying, which should be limited by overall network bandwidth.QuoteInternally, the sockets API is just a veneer over the netconn API. So something else is going on here.

author: dare
author_rank: Contributor
timestamp: February 10, 2023, 06:05:02 pm
content: Quote from: peter-h on February 10, 2023, 07:31:43 amOne more point on this: if useful, you can override LwIP's LWIP_DECLARE_MEMORY_ALIGNED macro to include explicit compiler directives, including alignment directives [__attribute__((aligned(4)))] and/or section directives [__attribute__ ((section ("LWIPBUFFERS")))].

author: peter-h
author_rank: Super Contributor
timestamp: February 10, 2023, 08:49:26 pm
content: QuoteYes, I know. It was a conscious hack. I have now done a small mod, in the form of a 200ms delay:Code:/* Create a new TCP connection handle */conn = netconn_new(NETCONN_TCP);if (conn!= NULL){/* Bind to port 80 (unless modified in config.ini) with default IP address */err = netconn_bind(conn, NULL, http_server_port);if (err == ERR_OK){// Put the connection into LISTEN statenetconn_listen(conn);while(true){// Accept the connection after netconn_listen// This function reports if there is some rx data (non blocking)err_t res = netconn_accept(conn, &newconn);if (res == ERR_OK){// Something arrived from the clientdebug_thread("http incoming connection");// Wait a bit for a complete request to arrive// This is a hack; ideally we should wait for end of each 'body'.osDelay(200);// Respond to the client requesthttp_server_serve(newconn);// Delete the connectionnetconn_delete(newconn);}}}else{debug_thread("cannot bind netconn");}}A better way would be to somehow get how much data has arrived and wait until that figure has stopped rising, for say 200ms.

author: peter-h
author_rank: Super Contributor
timestamp: February 12, 2023, 11:30:27 am
content: Just spent a couple of days playing with task priorities...I have a couple of tasks which use LWIP and which get suspended for a few seconds during TLS RSA/EC crypto. One (a primitive http server) uses Netconn and the other (a serial to TCP data copy process) uses sockets.I also have a number of tasks which don't do any networking and which run as they should, throughout. One of these copies a RAM buffer to some 7 segment LEDs, via an SPI LED controller. Experimentation of what priority these need is difficult but it looks like it needs to be at/above the tasks which invoke TLS. If their priority is 0 then TLS hangs them up as well.After much experimentation with RTOS priorities, this is what I found, and I wonder if it is right:TCP/IP applications (whether using the LWIP Netconn API or the LWIP socket API) should run at a priority equal to or lower than that of LWIP itself which [in this project] is osPriorityHigh (40). TCP/IP applications can be run with a priority all the way down to tskIDLE_PRIORITY (0).The exception is if TLS is in use. TLS does not yield to the RTOS; you get a solid CPU time lump of ~3 secs (STM 32F417, hardware AES only). TLS starts in the priority of the task which invokes it, but subsequent TLS-driven TCP/IP operations run at the priority of LWIP. So when TLS is doing the session setup crypto, tasks of a priority lower than LWIP get suspended. If this gap is an issue, the priority of the relevant tasks should be equal to LWIP's. Furthermore, due to the structure of LWIP, the priority of a task using it should not be higher than LWIP (24) since it might cause blocking on socket (or netconn) writes.Does this make sense?It looks like LWIP blocks all netconn and socket ops when TLS is using it. Is that possible?I am running with#define LWIP_TCPIP_CORE_LOCKING 0#define LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT 1#define SYS_LIGHTWEIGHT_PROT 1as this was found to have much better granularity for task switching. With LWIP_TCPIP_CORE_LOCKING=1 you end up with a crude mutex across the entire API call, which is fine in a single RTOS task.Thank you in advance for any pointers.

author: dare
author_rank: Contributor
timestamp: February 12, 2023, 04:49:37 pm
content: Quote from: peter-h on February 12, 2023, 11:30:27 amWhen you say a "solid lump" what exactly do you mean? Do you mean literally nothing else of equal priority gets to run? This shouldn't be happening. To wit: if you have two tasks running at the same priority, one that uses TLS and one that doesn't, and both tasks are purely computation bound (i.e. no blocking on synchronization objects), then you should see time-slicing between the two tasks at the interval of the system tick. If you're not seeing this, it means: 1) your other tasks are actually running at a lower priority than the TLS task, 2) the TLS task is holding a lock which it shouldn't, or 3) preemptive tasking has been disabled in your RTOS.QuoteI don't see how this is possible. Given the way the LwIP socket and netconn APIs work**, all TLS computation should be happening solely in the application thread. Furthermore, when that computation occurs, the application thread should not be holding any synchronization object that the LwIP tcpip thread needs. Therefore, you should never have a situation where priority is inherited from the tcpip task to the application task (i.e. where the OS boosts the priority of the app task because it happens to be holding a mutex that the tcpip task needs).** To expand on this a little further: the LwIP socket and netconn APIs (but not the raw API) are largely "down-call" APIs, meaning that interactions across the API happen by the API consumer calling a function, and LwIP performing some desired action and/or returning some data by the time that function returns. For the most part, there are no "up-calls" in these API, where LwIP calls back into the application's code.The only exception to this model is the optional netconn "callback" feature, where the LwIP task will call back into the application's code when a state change occurs on the connection (data available to read, space available to write, etc.). Although the LwIP docs do not make this clearat all, the application codeshould notdo any work in these callbacks. Rather, it should wake the application task, which would then take action to handle the state change.So, if you happen to be using this callback feature (e.g. if you are calling netconn_new_with_callback()) and you are doing work inside the callback function, this could be a source of your problems.QuoteFrankly, in your case, I think something more subtle is going on. All LwIP API calls end up being serialized, regardless of whether core locking is enabled. Its just that, with core locking, the overhead of that serialization is much less (a single mutex acquisition/release vs two task switches minimum). So enabling core locking should never make things worse unless the system is misconfigured in some way.

author: peter-h
author_rank: Super Contributor
timestamp: February 12, 2023, 05:27:44 pm
content: QuoteTasks of equal or higher prioritydorun through the TLS ops - unless they call LWIP. Then they get "hung up" for the ~3 seconds.I've been trying to trace this by waggling a GPIO pin and watching it on a scope and it appears that the task itself does run as it should (e.g. its main loop still runs at say 100Hz - if there is a osDelay(10) at the bottom of the loop) but calls to the socket API appear to be ineffective.I have not yet narrowed it down to whether lwip_read returns no data or the result of lwip_write disappears inside LWIP. But no data is actually lost.QuoteThat is what I understand, but what happens to priority when TLS makes a lwip_write or lwip_read API call? Is that LWIP code running in the calling task's priority?QuoteAFAIK this is not used.QuoteIt may well be.It could be that with core locking ON one needs more buffering around somewhere. I've spent many weeks on this and during that time have tried stuff like a huge increase in buffers (which the finished product could not run with) and it made no difference.

author: dare
author_rank: Contributor
timestamp: February 12, 2023, 06:51:45 pm
content: Quote from: peter-h on February 12, 2023, 05:27:44 pmlwip_write() ultimately calls netconn_write_partly() (LwIP sockets being a veneer on the netconn API). With core locking off, the netconn code creates an "API message", which captures the arguments to the write call. The API message includes a pointer to function that actually does the work of writing data to the connection, and a semaphore that is used to signal completion of the operation. The API message is then appended to the work queue of the tcpip task, after which the application task waits on the completion semaphore. The tcpip task dequeues the API message, calls the associated work function, and sets the semaphore, which in turn causes the application task to wake and ultimately return from the lwip_write() function.The same process happens for any other task that is using either the sockets or netconn APIs. Thus all LwIP API calls are serialized by means of the tcpip task work queue. Also, because the synchronization object is a semaphore, at no point is there an inheritance of priority between application task(s) and the lwip tcpip task (and certainly no point at which the tcpip task's priority would bedemotedto that of the app task).Given this, there's no obvious way that an app task running TLS crypto code would block or otherwise hinder the tcpip task in servicing requests from other tasks.But let me try another line of reasoning: how much outbound traffic is being generated by the TLS thread(s)? If I recall correctly, you are running with MEM_USE_POOLS = 0, MEM_LIBC_MALLOC = 0, MEMP_MEM_MALLOC = 1 and MEM_SIZE = 6KB. If a TLS thread is writing a lot of data, it could be dominating that 6K memory pool, which will cause other threads to have to wait when writing data to their connections. (Also remember that, if running as a TLS server, there is a point during the handshake where the server certificates are sent in a big chunk to the client. This can be a lot of data depending on how big your certificates are).QuoteWas this an increase in PBUF_POOL_SIZE or MEM_SIZE?

author: peter-h
author_rank: Super Contributor
timestamp: February 12, 2023, 09:36:08 pm
content: In the absence of clear experimental data I have gone back to core locking 1 (and undefined these two#define LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT 1#define SYS_LIGHTWEIGHT_PROT 1I see a 2x speedup on file downloads in the http server, despite it running in idle priority (0). Previously it would need a priority of 1 (above various other tasks) to achieve this. So yes core locking ON does seem to improve performance.QuoteIs that still the case with core locking ON?QuoteHmm, it was the latter, not the formerWith#define PBUF_POOL_BUFSIZE 1500 + PBUF_LINK_ENCAPSULATION_HLEN + PBUF_LINK_HLEN + 2increasing PBUF_POOL_SIZE from 4 to 5 of the above does amazingly remove the "TLS hanging" problem 99% of the time, so I went to 6.At a cost of ~3k of RAM. This is OK.I realise this is application(s) dependent but I am already running a pretty well fully loaded system.Thank you for this insight!

author: peter-h
author_rank: Super Contributor
timestamp: February 13, 2023, 09:25:03 pm
content: This is how I understand this worksThe RTOS is pre-emptive, with a 1ms tick. Even if a task runs in a loop without yielding to RTOS, it will still get pre-empted by tasks whose priority is same or higher.When an application makes an LWIP API call (Netconn or Sockets) the code initially runs at the priority of the calling task (whose priority can be anything from tskIDLE_PRIORITY (0) to LWIP's core priority which is osPriorityHigh (40)). LWIP then passes control to its core. The LWIP API is mutex protected so that only one calling task at a time can be active inside the API. This does not compromise multitasking performance because LWIP internally serialises the various requests anyway.Normally this scheme works well because LWIP runs fast, and the ETH 10/100 interface runs fast, but if a large packet - bigger than about 4k - is being sent out, LWIP can run out of buffers while waiting for something, and this will hold up other tasks whose priority is less than LWIP. This can happen regardless of whether the other tasks use the LWIP API.For tasks which use LWIP, and since these must not have a priority above LWIP, the only solution is to move smaller data blocks. However, if TLS is not used, this is rarely an issue.For tasks which do not use LWIP, a priority same as or greater than LWIP will enable them to run without gaps.The only time this has been observed as a problem is when TLS is in use. TLS performs crypto operations which for session setup can run for several seconds (RSA/EC crypto), and it transmits a lot of data. The result is that there is a block on the execution of tasks with a priority below LWIP's, for this duration.It could well be that nearly all weird issues are due to not enough buffers in various places, but I sure as hell cannot work out why the netconn-based http server gets hung up by the ~3 sec TLS activity. I've tried loads more buffers and it makes no difference.One thing I'd like to know is the relationship between MEM_SIZE and PBUF_POOL_SIZE and PBUF_POOL_BUFSIZE. This is a comment from my experiments from a year or so agoCode:// MEM_SIZE: the size of the heap memory. This is a statically allocated block. You can find it// in the .map file as the symbol ram_heap and you can see how much of this RAM gets used.// If MEMP_MEM_MALLOC=0, this holds just the PBUF_ stuff.// Empirically this needs to be big enough for at least 4 x PBUF_POOL_BUFSIZE.// 6k yields a good speed and going to 8k+ makes a minimal improvement. The main// factor affecting speed in the KDE is the poll period in ethernetif_input().// This value also limits the biggest block size sent out by netconn_write. With the// MEM_SIZE of 6k, the biggest block netconn_write will send out is 4k.I have 6k, 6 and MTU for the three, respectively, but examining the MEM_SIZE area (in the .map file as ram_heap) I see only 3k of it having been used, after some hours.As usual a lot of stuff online, mostly unclear.

author: dare
author_rank: Contributor
timestamp: February 14, 2023, 05:00:27 am
content: Quote from: peter-h on February 13, 2023, 09:25:03 pmClose, but not quite.First let me answer your earlier question about how LwIP works when core locking is enabled. Taking a similar scenario as before (an application task calling lwip_write()) the lwip code follows a nearly identical path as without core locking. Specifically, lwip_write() ultimately calls netconn_write_vectors_partly() which creates an API message structure that includes the data to be written, a work function pointer and a completion semaphore. Then the code calls tcpip_send_msg_wait_sem(), just as it did in the non-core locking case. However, at this point, instead of queuing the API message in the tcpip task's work queue (as it did in the non-core locking case), tcpip_send_msg_wait_sem() acquires the global "core lock" mutex and directly calls the work function itself (again, all on the app task). By acquiring the core lock, the code ensures that no other task, including the LwIP tcpip task itself, could be touching any of the shared data structures that hold the network state.The work function (lwip_netconn_do_write()) behaves pretty much identically in both the core locking and non-core locking modes. Its job is to take the application's data, append it onto the TCP send queue (possibly copying it in the process) and, if the time is right, initiate the sending of a TCP packet containing some or all of the data. When a packet is to be sent, the application's data is prepended with the appropriate TCP, IP and Ethernet headers, and the final Ethernet packet passed to the netif output function. Again, when core locking is enabled, all this happens on the application's task. The job of the netif output function is to queue the Ethernet packet for transmission by the MAC. For proper behavior, its design should be such that it never blocks. Thus, in the "sunny day" scenario (i.e. where there are plenty of buffers available) the only time the application thread will block is when it is waiting for the core lock. And since the LwIP code is designed to never block when the core lock is held, the amount of time an application task will spend waiting for the core lock should always be short.Okay, so then what happens in the non-sunny day case when you run out of buffers? In this situation, the work function discovers that there aren't enough buffers to all the application's data. If the socket/netconn happens to be in non-blocking mode, then function just stops where it is and returns an error back to the application telling it to try again. But if the netconn is in blocking mode, then by definition the API function cannot return until all the data has been consumed (or a fatal connection error has occurred). In this case, the work function (still running on the app task) places the netconn in a mode that says there is more application data waiting to be added to the TCP send queue. It then releases the core lock and waits on the completion semaphore. With the core lock released, the LwIP tcpip task is now free do its thing, one of which is to wake up periodically and check if there more buffers are available. When there are buffers available it proceeds to fill them with the application's data. Once all the data has been copied, the tcpip task sets the completion semaphore, which wakes up the application task, and causes it to return from the API write function.OK, with all that said, let me attempt to rewrite your summary of the way things work:FreeRTOS is pre-emptive, with a 1ms tick. Even if a task runs in a loop without yielding, it will still get pre-empted by other tasks whose priority is the same or higher.When an application makes an LwIP API call (Netconn or Sockets) the code initially runs on the calling application's task, at its set priority. [For core locking mode only] At some point during the API call the code will attempt to acquire the LwIP core lock mutex, and if necessary, will block waiting for another task to release it. However, since the LwIP code is designed to never block while holding the core lock, the wait time will be small, and in many cases 0. Once the core lock is acquired, the LwIP code running on the app task completes the requested API operation, releases the core lock and returns to the application code. In this way, the LwIP core lock serves to serialize all activities in the LwIP stack, so as to protect shared data structures.Normally this works well. However, if the system is busy, LwIP can run out of buffers. When this happens, the application task will block in the API call waiting for more buffers to be available. The LwIP tcpip task will then periodically check the state of the buffer pool, and once buffers are available, it will complete the outstanding API work and wake the application task.When buffers are low, other tasks calling the LwIP API concurrently will not block unless they too need buffers to complete their work. Compute-bound tasks which are not calling LwIP (e.g. those performing heavy crypto) will not affect activity in LwIP regardless of buffer availabilityunlessthe priority of the compute-bound tasks is higher than that of the LwIP app tasks, or the LwIP tcpip task itself. For best communications performance, the LwIP tcpip task should run at a high priority, the LwIP application tasks should run at a medium priority and compute-bound tasks should run at a lower priority.Unfortunately, because most LwIP applications need buffers to do their work, running out of buffers can bring everything to a near stand-still. Generally, the only solution is either to increase the size of the buffer pool or throttle the overall communication rate (send, receive or both) of the LwIP application tasks.QuoteGiven your configuration, I think the simple answer is that MEM_SIZE governs the number of buffers available for outgoing data (i.e. data your app send over the network) and PBUF_POOL_SIZE governs the buffers available for incoming data (i.e. packets you receive from the network). Setting MEM_SIZE too small will induce the kind of buffer starvation/write blocking behavior described above. Setting PBUF_POOL_SIZE too small will result in inbound packets being dropped in the low-level input function.

author: dare
author_rank: Contributor
timestamp: February 14, 2023, 05:24:43 am
content: Regarding the particular behavior you're seeing, I see that you have set TCP_SND_BUF to 4 * TCP_MSS. That works out to 4 * (1500 - 40) = 5840. TCP_SND_BUF represent the maximum amount of application data that can be written to a particular the TCP send queue. That number is awfully close to your MEM_SIZE value. What this means is that a single TCP connection can dominate the buffers available in your memory heap if it happens to be writing faster than other tasks.Perhaps this is what is happening with your TLS task. TLS sends a huge chunk of data (the server certificates) to the peer during the handshake. Because your window size is 2*TCP_MSS, getting all that data to the peer likely requires multiple round trips. Furthermore, LwIP + FreeRTOS isn't necessarily fair about dolling out buffers when they become available. So it may be that your TLS task is getting more than its fair share.Try setting TCP_SND_BUF to 2*TCP_MSS and see if that helps. I would also make sure that all app tasks that use LwIP are running at the same priority. I might even go so far as to set your TLS task tolowerpriority than your other tasks.

author: peter-h
author_rank: Super Contributor
timestamp: February 15, 2023, 07:46:08 am
content: I cannot thank you too much,dare, for these amazing and illuminating posts.BTW I did more tests, with a) LWIP_TCPIP_CORE_LOCKING ON and b) LWIP_TCPIP_CORE_LOCKING OFF (and with LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT and SYS_LIGHTWEIGHT_PROT ON as apparently required if calling the API from multiple tasks) and the only difference I can reproduce is that the former delivers better file transfer performance for a given fairly small number of PBUFs. But this is transferfromLWIP so presumably this increase is related to being able to buffer more ACKs. So I have gone back to LWIP_TCPIP_CORE_LOCKING ON. As you said it should not make much difference because LWIP internally serialises the processing of the messages anyway (it runs onlyoneRTOS task).QuoteThis is so poorly documented in the various online sources. I did find, empirically, that MEM_SIZE directly limits the biggest packet which can be sent in one go. What happens if you try to send a bigger one I didn't investigate because I've been usingCode:netconn_write(conn, PAGE_HEADER_ALL, strlen((char*)PAGE_HEADER_ALL), NETCONN_COPY);and not checking the return code. In the context there was little one could do with it. A MEM_SIZE=6k allowed about 4k to be sent out. So MEM_SIZE is really abouttransmit.Various sources suggest the PBUFs are used forrx only, while others suggest they are used for both rx and tx. I could not see the point of using this "list of buffers" system for tx because tx can be done by taking the caller's packet and repackaging it (presumably using some buffer temporarily allocated in MEM_SIZE) into one or more packets which are sent down to the PHY layer. In transmit mode, LWIP has total control. OTOH TCP packets are ACKed so presumably the PBUFs are used for these too and there would lie another bottleneck; that may be a reason for having a larger number of smaller PBUFs (the original ST port of LWIP used 512 byte PBUFs).QuoteIt was already set to 2xTCP_MSS following your earlier suggestion.QuoteI asked the guy who did the TLS integration on this product. It was empirically determined that TLS sends out about 1k during the setup. For the size of subsequent data packets, one can negotiate the size, and we have set this to 4k. So yeah one comes back to your 4k. This relates to TLS needing two 16k buffers by definition (16k rx and 16k tx) but if you control one or both ends then these can be reduced. With an HTTPSclientthe rx buffer must still be 16k but the tx buffer can be smaller because you control what is sent out, and thus save 12k of RAM.For background: Currently TLS is working with a statically allocated block of 48k in which it runs its private heap; this seems sufficient for everything that's been tested, but unfortunately this is not (and cannot be) deterministic. It depends on what crypto suites get negotiated. AES was using quite a lot of RAM (the TLS implementation was actually pretty fast, at 800kbytes/sec) and moving it to the 32F417's hardware AES saved 12k. But other savings are much smaller and the 417 doesn't do RSA/EC hence all this hassle with the 3 sec hangs. We have not tested DES/3DES which could be moved to 417 hardware, but I've not been able to find out if anybody out there might still be using it, and there is a proposal for new TLS to drop DES. I think this is dangerous because e.g. it was found that some root certificates (some of these are old e.g. 2005) are signed with a "supposedly deprecated" hash and if you tried to be clever and removed support for obsolete hashes because you read on the internet that they should not be used, you would have a problem! So the guy ported our TLS code to a win32 executable which can be run on a connection which isn't working and will tell you directly why not. The problem with doing crypto in hardware is that it prevents one using one of the chinese copies of the 417 which are just a 407 (no crypto).I've tried increasing MEM_SIZE to silly values and it doesn't fix the hanging issue.I've tried the priority adjustment and it makes no difference. Currently all "internet" tasks run at idle (0).The main task which was getting hung up by TLS is now running OK due to pool size = 6 (was 4)Code:/* ---------- Pbuf dynamically allocated buffer blocks ----------** These settings make little difference to performance, which is dominated by* the low_level_input poll period. These PBUFs relate directly to the netconn API netbufs.** PBUF_POOL_SIZE: the number of buffers in the pbuf pool.* Statically allocated, so each increment frees up 1.5k of RAM. Any value under 4 slows* down the data rate a lot.* 12/2/23 PH 4 -> 6 fixes TLS blocking of various RTOS tasks (5 almost does).* The most this can go to with TLS still having enough free RAM for its 48k block is ~9.*/#define PBUF_POOL_SIZE 6/* PBUF_POOL_BUFSIZE: the size of each pbuf in the pbuf pool.* The +2 is to get a multiple of 4 bytes so the PBUFs are 4-aligned (for memcpy_fast())* Probably the +2 is not needed because the PBUFs are 4-aligned anyway in the LWIP code.*/#define PBUF_POOL_BUFSIZE 1500 + PBUF_LINK_ENCAPSULATION_HLEN + PBUF_LINK_HLEN + 2The remaining task that stillisgetting hung up by TLS is an http server which uses the netconn API and nothing I do seems to fix that, not even a priority of 40 (same as LWIP) or a huge increase in the buffers, but it doesn't matter because it is intended for local config only. I would never let a user run a server on an open port anywayI am very sure that its RTOS task isn't actually hanging; it is hanging on the LWIP API calls. That in turn suggests insufficient buffers somewhere.FWIW my lwipopts.h file is below, in case anybody is digging around here in years to comeCode:/********************************************************************************* @file LwIP/LwIP_HTTP_Server_Netconn_RTOS/Inc/lwipopts.h* @author MCD Application Team* @brief lwIP Options Configuration.********************************************************************************This sort of explains the memory usage*https://lwip-users.nongnu.narkive.com/dkzkPa8l/lwip-memory-settings*https://www.cnblogs.com/shangdawei/p/3494148.html*https://lwip.fandom.com/wiki/Tuning_TCP*https://groups.google.com/g/osdeve_mirror_tcpip_lwip/c/lFYJ7Fw0Cxg*ST UM1713 document gives an overview of integrating all this.*****7/7/22PHMEM_SIZE set to 5k (was 10k). Only ~1.5k is used.*13/7/22PHMEMP_MEM_MALLOC=1, MEM_SIZE=16k.*14/7/22PHMEMP_MEM_MALLOC=0; 1 was unreliable. 8 x 512 byte buffers now.*6/8/22PH4 x MTU buffers for RX. Done partly for EditGetData().*MEM_SIZE=6k. 5k is used - see .map file for ram_heap address.*Sizes of various static RAM structures determined experimentally.* 17/12/22PHLWIP_TCP_KEEPALIVE=1 (for ethser)* 27/1/23PHIP_SOF_BROADCAST etc added, later commented-out.* 10/2/23PHPBUF_POOL_BUFSIZE 4-aligned.* 12/2/23PHPBUF_POOL_SIZE = 6.*********/#ifndef __LWIPOPTS_H__#define __LWIPOPTS_H__/*** NO_SYS==1: Provides VERY minimal functionality. Otherwise,* use lwIP facilities.*/#define NO_SYS0// Flag to make LWIP API thread-safe. The netconn and socket APIs are claimed// to be thread-safe anyway. The raw API is never thread-safe.// A huge amount of online discussion on this topic; most of it unclear, but// ON (1) seems to be recommended, as being more efficient.#define LWIP_TCPIP_CORE_LOCKING1// If LWIP_TCPIP_CORE_LOCKING=0 then these two need to be 1// See https://www.nongnu.org/lwip/2_1_x/multithreading.html//#define LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT 1//#define SYS_LIGHTWEIGHT_PROT 1// This places more objects into the static block defined by MEM_SIZE.// Uses mem_malloc/mem_free instead of the lwip pool allocator.// MEM_SIZE now needs to be increased by about 10k.// It doesn't magically produce extra memory, and causes crashes.// There is also a performance loss, apparently. AVOID.#define MEMP_MEM_MALLOC0//NC: Need for sending PING messages by keepalive#define LWIP_RAW1#define DEFAULT_RAW_RECVMBOX_SIZE4// For ETHSER#define LWIP_TCP_KEEPALIVE1/*-----------------------------------------------------------------------------*//* LwIP Stack Parameters (modified compared to initialization value in opt.h) -*//* Parameters set in STM32CubeMX LwIP Configuration GUI -*//*----- Value in opt.h for LWIP_DNS: 0 -----*/#define LWIP_DNS 1/* ---------- Memory options ---------- *//* MEM_ALIGNMENT: should be set to the alignment of the CPU for whichlwIP is compiled. 4 byte alignment -> define MEM_ALIGNMENT to 4, 2byte alignment -> define MEM_ALIGNMENT to 2. */#define MEM_ALIGNMENT 4/** MEM_SIZE: the size of the heap memory. This is a statically allocated block. You can find it* in the .map file as the symbol ram_heap and you can see how much of this RAM gets used.* If MEMP_MEM_MALLOC=0, this holds just the PBUF_ stuff.* If MEMP_MEM_MALLOC=1 (which is not reliable) this greatly expands and needs 16k+.* Empirically this needs to be big enough for at least 4 x PBUF_POOL_BUFSIZE.* This value also limits the biggest block size sent out by netconn_write. With a MEM_SIZE* of 6k, the biggest block netconn_write (and probably socket write) will send out is 4k.* This setting is mostly related to outgoing data.*/#define MEM_SIZE (6*1024)// MEMP_ structures. Their sizes have been determined experimentally, by// increasing them and seeing free RAM changing./* MEMP_NUM_PBUF: the number of memp struct pbufs. If the applicationsends a lot of data out of ROM (or other static memory), thisshould be set high. *///NC: Increased to 20 for ethser#define MEMP_NUM_PBUF 20// each 1 is 20 bytes of static RAM/* MEMP_NUM_UDP_PCB: the number of UDP protocol control blocks. Oneper active UDP "connection". */#define MEMP_NUM_UDP_PCB 6// each 1 is 32 bytes of static RAM/* MEMP_NUM_TCP_PCB: the number of simultaneously active TCPconnections. *///NC: Increased to 20 for ethser#define MEMP_NUM_TCP_PCB 20// each 1 is 145 bytes of static RAM//NC: Have more sockets available. Is set to 4 in opt.h#define MEMP_NUM_NETCONN10/* MEMP_NUM_TCP_PCB_LISTEN: the number of listening TCPconnections. *///NC: Increased to 20 for ethser#define MEMP_NUM_TCP_PCB_LISTEN 20// each 1 is 28 bytes of static RAM/* MEMP_NUM_TCP_SEG: the number of simultaneously queued TCPsegments. */// Was 8; increased to 16 as it improves ETHSER reliability when running// HTTP server#define MEMP_NUM_TCP_SEG 16 // each 1 is 20 bytes of static RAM/* MEMP_NUM_SYS_TIMEOUT: the number of simulateously activetimeouts. */#define MEMP_NUM_SYS_TIMEOUT 10// each 1 is 16 bytes of static RAM/* ---------- Pbuf dynamically allocated buffer blocks ----------** These settings make little difference to performance, which is dominated by* the low_level_input poll period. These PBUFs relate directly to the netconn API netbufs.** PBUF_POOL_SIZE: the number of buffers in the pbuf pool.* Statically allocated, so each increment frees up 1.5k of RAM. Any value under 4 slows* down the data rate a lot.* 12/2/23 PH 4 -> 6 fixes TLS blocking of various RTOS tasks (5 almost does).* The most this can go to with TLS still having enough free RAM for its 48k block is ~9.*/#define PBUF_POOL_SIZE 6/* PBUF_POOL_BUFSIZE: the size of each pbuf in the pbuf pool.* The +2 is to get a multiple of 4 bytes so the PBUFs are 4-aligned (for memcpy_fast())* Probably the +2 is not needed because the PBUFs are 4-aligned anyway in the LWIP code.*/#define PBUF_POOL_BUFSIZE 1500 + PBUF_LINK_ENCAPSULATION_HLEN + PBUF_LINK_HLEN + 2/* ---------- TCP options ---------- */#define LWIP_TCP 1#define TCP_TTL 255/* Controls if TCP should queue segments that arrive out oforder. Define to 0 if your device is low on memory. */#define TCP_QUEUE_OOSEQ 0/* TCP Maximum segment size. */#define TCP_MSS (1500 - 40)/* TCP_MSS = (Ethernet MTU - IP header size - TCP header size) *//* TCP sender buffer space (bytes). */// Reduced from 4*MSS to leave more room for TX packets in the LWIP heap (MEM_SIZE).#define TCP_SND_BUF (2*TCP_MSS)// no effect on static RAM/* TCP_SND_QUEUELEN: TCP sender buffer space (pbufs). This must be at leastas much as (2 * TCP_SND_BUF/TCP_MSS) for things to work. */// Was 2*; increased to 4* as it improves ETHSER reliability when running// HTTP server#define TCP_SND_QUEUELEN (4* TCP_SND_BUF/TCP_MSS) // (2* TCP_SND_BUF/TCP_MSS)/* TCP advertised receive window. */// Should be less than PBUF_POOL_SIZE * (PBUF_POOL_BUFSIZE - protocol headers)#define TCP_WND (2*TCP_MSS)// no effect on static RAM/* ---------- ICMP options ---------- */#define LWIP_ICMP1/* ---------- DHCP options ---------- */#define LWIP_DHCP 1/* ---------- UDP options ---------- */#define LWIP_UDP 1#define UDP_TTL 255// These are build flags which disable the support for the SOF_BROADCAST option on raw and UDP PCBs// Commented-out because changing these requires a recompilation, and an application which receives// broadcast packets may one day be necessary (set g_eth_multi=true to disable the packet filter in// ethernetif.c)//#define IP_SOF_BROADCAST 1//#define IP_SOF_BROADCAST_RECV 1/* ---------- Statistics options ---------- */#define LWIP_STATS 0/* ---------- link callback options ---------- *//* LWIP_NETIF_LINK_CALLBACK==1: Support a callback function from an interface* whenever the link changes (i.e., link down)* 8/2022 this is done from the low_level_input RTOS task.*/#define LWIP_NETIF_LINK_CALLBACK 0/*------------------------------------------------ Checksum options ------------------------------------------------*//*The STM32F4xx allows computing and verifying the IP, UDP, TCP and ICMP checksums by hardware:- To use this feature let the following define uncommented.- To disable it and process by CPU comment the the checksum.*/#define CHECKSUM_BY_HARDWARE#ifdef CHECKSUM_BY_HARDWARE/* CHECKSUM_GEN_IP==0: Generate checksums by hardware for outgoing IP packets.*/#define CHECKSUM_GEN_IP 0/* CHECKSUM_GEN_UDP==0: Generate checksums by hardware for outgoing UDP packets.*/#define CHECKSUM_GEN_UDP 0/* CHECKSUM_GEN_TCP==0: Generate checksums by hardware for outgoing TCP packets.*/#define CHECKSUM_GEN_TCP 0/* CHECKSUM_CHECK_IP==0: Check checksums by hardware for incoming IP packets.*/#define CHECKSUM_CHECK_IP 0/* CHECKSUM_CHECK_UDP==0: Check checksums by hardware for incoming UDP packets.*/#define CHECKSUM_CHECK_UDP 0/* CHECKSUM_CHECK_TCP==0: Check checksums by hardware for incoming TCP packets.*/#define CHECKSUM_CHECK_TCP 0/* CHECKSUM_CHECK_ICMP==0: Check checksums by hardware for incoming ICMP packets.*/#define CHECKSUM_GEN_ICMP 0#else/* CHECKSUM_GEN_IP==1: Generate checksums in software for outgoing IP packets.*/#define CHECKSUM_GEN_IP 1/* CHECKSUM_GEN_UDP==1: Generate checksums in software for outgoing UDP packets.*/#define CHECKSUM_GEN_UDP 1/* CHECKSUM_GEN_TCP==1: Generate checksums in software for outgoing TCP packets.*/#define CHECKSUM_GEN_TCP 1/* CHECKSUM_CHECK_IP==1: Check checksums in software for incoming IP packets.*/#define CHECKSUM_CHECK_IP 1/* CHECKSUM_CHECK_UDP==1: Check checksums in software for incoming UDP packets.*/#define CHECKSUM_CHECK_UDP 1/* CHECKSUM_CHECK_TCP==1: Check checksums in software for incoming TCP packets.*/#define CHECKSUM_CHECK_TCP 1/* CHECKSUM_CHECK_ICMP==1: Check checksums by hardware for incoming ICMP packets.*/#define CHECKSUM_GEN_ICMP 1#endif/*-------------------------------------------------------- Sequential layer options --------------------------------------------------------*//*** LWIP_NETCONN==1: Enable Netconn API (require to use api_lib.c)*/#define LWIP_NETCONN 1/*---------------------------------------------- Socket options ----------------------------------------------*//*** LWIP_SOCKET==1: Enable Socket API (require to use sockets.c)*/#define LWIP_SOCKET 1/*---------------------------------------------- httpd options ----------------------------------------------*//** Set this to 1 to include "fsdata_custom.c" instead of "fsdata.c" for the* file system (to prevent changing the file included in CVS) */#define HTTPD_USE_CUSTOM_FSDATA 0/*------------------------------------------- OS options -------------------------------------------*/#define TCPIP_THREAD_NAME "TCP/IP"#define TCPIP_THREAD_STACKSIZE 4096#define TCPIP_MBOX_SIZE 6#define DEFAULT_UDP_RECVMBOX_SIZE 6#define DEFAULT_TCP_RECVMBOX_SIZE 6#define DEFAULT_ACCEPTMBOX_SIZE 6#define DEFAULT_THREAD_STACKSIZE 512#define TCPIP_THREAD_PRIO osPriorityHigh// should be >= that of any TCP/IP apps#define LWIP_DEBUG 1/*#define IP_DEBUG LWIP_DBG_ON#define DHCP_DEBUG LWIP_DBG_OFF#define UDP_DEBUG LWIP_DBG_ON#define SOCKET_DEBUG_LWIP_DBG_ON//#define ICMP_DEBUG LWIP_DBG_ON|LWIP_DBG_TRACE//#define NETIF_DEBUG LWIP_DBG_OFF#define LWIP_DBG_TYPES_ON (LWIP_DBG_TRACE|LWIP_DBG_STATE)*/#define LWIP_SO_RCVTIMEO1#define LWIP_NETIF_HOSTNAME1#define SO_REUSE 1// Defining these produces various errors//#define LWIP_IPV61//#define LWIP_IPV6_DHCP61/*// TODO#ifdef LWIP_DEBUG#define MEMP_OVERFLOW_CHECK ( 1 )#define MEMP_SANITY_CHECK ( 1 )#define MEM_DEBUG LWIP_DBG_OFF#define MEMP_DEBUG LWIP_DBG_OFF#define PBUF_DEBUG LWIP_DBG_ON#define API_LIB_DEBUG LWIP_DBG_ON#define API_MSG_DEBUG LWIP_DBG_ON#define TCPIP_DEBUG LWIP_DBG_ON#define NETIF_DEBUG LWIP_DBG_ON#define SOCKETS_DEBUG LWIP_DBG_ON#define DEMO_DEBUG LWIP_DBG_ON#define IP_DEBUG LWIP_DBG_ON#define IP6_DEBUG LWIP_DBG_ON#define IP_REASS_DEBUG LWIP_DBG_ON#define RAW_DEBUG LWIP_DBG_ON#define ICMP_DEBUG LWIP_DBG_ON#define UDP_DEBUG LWIP_DBG_ON#define TCP_DEBUG LWIP_DBG_ON#define TCP_INPUT_DEBUG LWIP_DBG_ON#define TCP_OUTPUT_DEBUG LWIP_DBG_ON#define TCP_RTO_DEBUG LWIP_DBG_ON#define TCP_CWND_DEBUG LWIP_DBG_ON#define TCP_WND_DEBUG LWIP_DBG_ON#define TCP_FR_DEBUG LWIP_DBG_ON#define TCP_QLEN_DEBUG LWIP_DBG_ON#define TCP_RST_DEBUG LWIP_DBG_ON#define PPP_DEBUG LWIP_DBG_OFF#define LWIP_DBG_TYPES_ON (LWIP_DBG_ON|LWIP_DBG_TRACE|LWIP_DBG_STATE|LWIP_DBG_FRESH|LWIP_DBG_HALT)#endif*/#endif /* __LWIPOPTS_H__ */

author: dare
author_rank: Contributor
timestamp: February 15, 2023, 06:35:48 pm
content: Quote from: peter-h on February 15, 2023, 07:46:08 amWell, at the level of the LwIP API you're not sending a big "packet" per se, but rather just a big chuck of data. Regardless of how big a chuck you write, the actual size of the outgoing packets will be limited by the TCP_MSS value. And as I described earlier, if you exhaust the MEM_SIZE heap while writing a big chunk your task will block in the write call.QuoteAh, indeed. Sorry, I lost track of our earlier conversation.QuoteDES/3DES is dead, dead, dead. No need to even think about it.I'm assuming the other end of the TLS connection is going to be a relatively standard server/desktop class TLS implementation. In that case, I'd build the TLS code such that it only supports TLS 1.3, which only allows AES and ChaCha20-Poly1305 anyway. Indeed, I'd go so far as to limit the cipher suites to just TLS_AES_128_GCM_SHA256, or TLS_AES_128_CCM_SHA256 if you can get away with it. No need for anything else.QuoteMaybe this is easier said than done, but it seems like its time to get a debugger on it and see what the threads are doing when its hung.

author: peter-h
author_rank: Super Contributor
timestamp: February 16, 2023, 04:23:08 am
content: QuoteCan one be 100% sure? There are loads of servers out there, hosting highly public stuff like PHP-BB communities, which have not been patched for 20 years. One I know got hit recentlyThe box I am doing is for industrial applications and that area is notorious for "it if works, leave it". Also after a few years the person who set it up has usually left and most coders (especially server-side - something I am quite involved with spec-wise and management-wise, although I don't do the actual config and coding) never document anything. It is only when something breaks because some certificate has expired that... there is panic. And money...QuoteYou will have the last laugh. I did some GPIO waggle debug and found it was getting hung by TLS's filesystem reading of the certificates file. It was getting hung because the HTTP server also displays a file directoryFatFS obviously needs mutex protection for writing (FAT12 in this case) although it isn't obvious that a mutex is needed for reading (I have one though). Re-entrancy is an option on FatFS but if you enable it, you cannot run FatFS before the RTOS starts so e.g. you cannot read a config file which selects which RTOS tasks to run.Another interesting point, for which I can't find a clear explanation, is related to FreeRTOS#define configUSE_PREEMPTION 1#define configUSE_TIME_SLICING 1and whether the following is correct:The RTOS is not pre-emptive on a time slice basis at the same priority, so a process which is just a loop will get nearly all CPU time unless there are higher priority tasks running.The question is whether "same" is correct. I think if you have a load of tasks at priority 10, and all with awhile (true) {}loop, the RTOS will switch them at 1kHz and each one will get 1ms. This behaviour was changed in FreeRTOS some years ago, from what I can find.Obviously hanging like that is dumb and if you have nothing to do thenwhile (true){osDelay(1);}is much better because the RTOS can move on right away. And it can run lower priority tasks too, whereaswhile (true){taskYield();}enables it to run same or higher priority tasks only.

author: peter-h
author_rank: Super Contributor
timestamp: March 07, 2023, 04:38:28 pm
content: Not quite LWIP but MbedTLS runs on top of LWIP, usually.This turned up on the MbedTLS mailing list, and I am posting it here so somebody can have a laugh at the irony of "security" in the embedded sphereTo: mbed-tls@lists.trustedfirmware.orgSubject: [mbed-tls] security issue in mbedtls 3.30From: Avi Epstein via mbed-tls <mbed-tls@lists.trustedfirmware.org>Date: Tue, 07 Mar 2023 16:02:22 -0000security issue in mbedtls 3.30 in the release notes:"An adversary with access to precise enough information about memoryaccesses (typically, an untrusted operating system attacking a secureenclave) could recover an RSA private key after observing the victimperforming a single private-key operation if the window size used for theexponentiation was 3 or smaller. Found and reported by Zili KOU,Wenjian HE, Sharad Sinha, and Wei ZHANG. See "Cache Side-channel Attacksand Defenses of the Sliding Window Algorithm in TEEs" - Design, Automationand Test in Europe 2023."was this issue solved in this version?--mbed-tls mailing list -- mbed-tls@lists.trustedfirmware.orgTo unsubscribe send an email to mbed-tls-leave@lists.trustedfirmware.org

author: dare
author_rank: Contributor
timestamp: March 07, 2023, 06:59:17 pm
content: Quote from: peter-h on March 07, 2023, 04:38:28 pmI appears that this paper has yet to be published. However, based on the abstract, the described attack mode has nothing to do with the class of processors being discussed here. In particular, this is an attack against systems with Trusted Execution Environments (TEEs), e.g. Intel SGX or ARM TrustZone. And as with most such attacks, this one appears to be limited to situations where adversarial code is running on the target system itself, which is typically only seen in environments such as mobile phones and virtual machine services. Monolithic embedded applications such as those that run on small STM32 parts are not subject to this mode of attack.That said, side channel attacks (where secret data are leaked via seemly innocuousness means, such as EMF, power use fluctuations, or even sound) are very real and devilishly hard to avoid. Indeed, the laptop you're reading this on is probably leaking all sorts of things right now:https://www.cse.wustl.edu/~roger/566S.s21/09065580.pdf. Because of this, I believe that evidence of a side channel attack involving Mbed TLS is not really an indictment of IoT security as a whole as much as it is an indication of how hard security is in general.You have to keep this in perspective, though. It takes a lot of work to mount a practical side-channel attack against a monolithic embedded application. And in most cases this requires close proximity to the victim device. So side channel attacks generally are less urgent to defend against than, say, remote code execution attacks resulting from simple coding bugs.

author: peter-h
author_rank: Super Contributor
timestamp: March 07, 2023, 08:52:18 pm
content: Thast's an interesting paper (on the laptop stuff) but what happened to Van Eck? The old CRTs were great for that because the scan coils and grid drives generated tons of RF (I used to design CRT drive boards in the 1980s) and LCDs obviously work differently, but I am 100% sure you can recover a laptop screen image from a great distance.I think most side channel attacks rely on knowing the structure of the software running on the machine. If open source, this is trivial, so if you have say Apache running on one VM on some server, you know where to look.With embedded boxes, these are rarely open source so you have an uphill job right at the start. Key leakage is still possible (as you note, by supply current measurement, Vcc manipulation, etc, and smartcard chips are designed to resist that stuff, and you are free to put one of these in your IOT box if needed) but with no knowledge of the software, where do you start? Breaking the security fuse protection to start with, then disassembly...But what makes me laugh is that there is no security whatsoever without physical security, and 99.9% of the time in the IOT sphere you have no physical security. Mostly, you can access the actual box, get into some boot loader or whatever, replace the TLS certificate store, and if you install a fake DNS server on the LAN then you are in.There is stuff like secure Modbus, which is completely pointless since all interconnected boxes are usually on one LAN, with no physical security...I would bet that for remote code execution to be viable it would need to be applied against known open source software which you know is used inside the box - such as LWIP or MbedTLS. But it would still be hard work, probably needing to be preceeded by breaking the CPU security and disassembling a load of code.What does one do in applications where you do want secure comms for very good known reasons but you don't want a whole PC-like server in there? For example an electricity substation which you know is going to get remotely attacked by Russia? This is a decades-old problem which pre-dates modern over-internet hacking (look up the GI74 protocol). "IOT" is no good for that because the boxes are rarely if ever patched for security reasons. It is a good MbedTLS application - in theory but absolutely not in practice.

author: peter-h
author_rank: Super Contributor
timestamp: May 02, 2023, 02:38:30 pm
content: Would anyone know if LWIP or TLS heap manager has this bughttps://www.eevblog.com/forum/programming/help-needed-with-some-heap-test-code/msg4844144/#msg4844144

author: peter-h
author_rank: Super Contributor
timestamp: May 15, 2023, 02:51:05 pm
content: Apparently it does not.

author: Delboy
author_rank: Newbie
timestamp: June 13, 2023, 05:34:33 pm
content: I learned more about LWIP reading this thread than I knew using LWIP for the past 10 years. Thanks guys.Can anyone recommend a link or docs on how to integrate Mbedtls with LWIP using the netconn interface. I have it kind of working, but not 100% reliable, any hints would be great.

author: peter-h
author_rank: Super Contributor
timestamp: June 13, 2023, 08:33:20 pm
content: I know little about this but AFAIK MbedTLS calls LWIP via the BSD socket interface, not via the Netconn interface.Netconn is specific to LWIP but MbedTLS is usable with other TCP/IP stacks.Almost nobody is using Netconn. I wrote a whole (simple) HTTP server using the Netconn API but found almost no help with it. Should have used the socket API instead. It is built on top of the Netconn API anyway.Getting MbedTLS working properly is a long job - on top of getting LWIP working properlyQuick demos are easy enough...