W5500 ethernet... how in the hell it sends 10GB per month?! - Page 1
Forum section: electronics / projects

author: honeybadger
author_rank: Contributor
timestamp: October 23, 2020, 09:58:20 pm
content: Hi,I have a "smart home" (I hate that term) project which needs internet connectivity.It uses a common W5500 Ethernet controller and it works just fine.Every 5 seconds it send ~500 characters long text string message to remote server through HTTP POST.This is roughly 250MB of useful data per month.Surprise Motherf*cker!It eats through 2GB data plan in a week. It needs roughly 10GB per month!How is this possible? Where is the 9,75GB per month coming from? Is the HTTP POST really that data hungry with small frequent payload data?Is there anything I can do to lower the data usage?* I can not make the data string smaller.* I need the 5s refresh rate. OK maybe not 5s, 10s would be also fine but no more.* I can not cache the data and send a big package every a minute or so - not enough memory.* It works with a standard Arduino Ethernet lib.* It does not sends the same data 100x, nothing like that. I do not touch the W5500 between the 5s period.

author: magic
author_rank: Super Contributor
timestamp: October 23, 2020, 10:05:31 pm
content: tcpdump

author: ejeffrey
author_rank: Super Contributor
timestamp: October 23, 2020, 10:21:41 pm
content: Have you watched with wireshark to see what is actually happening? Are you sure there isn't something else on the network using the bandwidth? Maybe your ISP is counting wrong?Overhead shouldn't be more than 4x for this scenario. Assuming you are making a new HTTP connection for every request there will be ~8 total TCP packets with under 500 bytes total headers/framing. In addition you will have the 500 bytes of data, a few hundred bytes in HTTP headers, and a few hundred bytes in a minimal response.Is the arduino ethernet library using multi-byte SPI transfers? I have never used it but I have used the wiznet ioLibrary. It has 4 functions you need to implement: read/write byte (or word? I don't remember) and read/write buffer. If you don't provide an implementation for the latter, it falls back to doing single byte SPI transactions. It is possible that the wiznet chip handles this poorly by doing a whole bunch of 1 byte packets. That would result in the sort of behavior you are seeing.

author: honeybadger
author_rank: Contributor
timestamp: October 23, 2020, 10:32:04 pm
content: Colleague will help me with the wireshark next week. I am definitely not a network guy.There is not anything else stealing the bandwidth. The "thing" with W5500 is connected directly to 4G router. I guess the mobile provider is counting the data correctly.Not sure about the multi-byte SPI transfer. How can I find it out?

author: ejeffrey
author_rank: Super Contributor
timestamp: October 23, 2020, 10:40:45 pm
content: With an oscilloscope or logic analyzer on the SPI bus.tcpdump/wireshark aren't hard to use. You don't need to be a networking expert or anything. If you write applications that communicate over a network then you should consider that essential knowledge.

author: ataradov
author_rank: Super Contributor
timestamp: October 24, 2020, 12:05:11 am
content: This was a very common problem with GPRS modems in the past. Wireless providers would have billing setup in 64 Kbyte minimum, so even if you send 2 bytes per hour. you will still get billed for 64 K.This got solved with introduction of data-only SIM cards and corresponding billing plans.I'm not saying it is the case here, but definitely could be.tcpdump/wireshark will tell for sure.

author: free_electron
author_rank: Super Contributor
timestamp: October 24, 2020, 12:22:43 am
content: a single ethernet frame is what these days ? 64bytes to1518 bytes. make sure you use smallest frame possible and not waste 1518 bytes to send 3 bits ...

author: NiHaoMike
author_rank: Super Contributor
timestamp: October 24, 2020, 01:20:03 am
content: Quote from: honeybadger on October 23, 2020, 09:58:20 pmHave you considered LoRa?

author: girishv
author_rank: Regular Contributor
timestamp: October 24, 2020, 02:59:04 am
content: Quote from: honeybadger on October 23, 2020, 09:58:20 pmIf you can make changes on server side to send delivery receipts and handle the delivery errors on firmware level, you can consider using UDP.Furthermore, if you are sending repetitive data, you may ignore the delivery errors.

author: ejeffrey
author_rank: Super Contributor
timestamp: October 24, 2020, 03:34:28 am
content: Quote from: girishv on October 24, 2020, 02:59:04 amDo not do that. Unless you are a networking wizard (which the OP has assured us they are not) you should just pretend UDP doesn't exist. That goes doubly so if you are talking about routing over the public internet like it appears the OP is doing.HTTP is a perfectly fine protocol for this application, something is just going wrong. Randomly changing to a different protocol without figure out what is going on is a waste of time, especially one with the possibility to create a whole host of other problems.The only sensible course of action is to use a packet sniffer to see what is actually happening. Either your networking library/offload chip is doing something silly or your ISP is counting data wrong.

author: jbb
author_rank: Super Contributor
timestamp: October 24, 2020, 07:32:40 am
content: +1 for wiresharkI wouldn’t be surprised if the service provider is rounding up per billing period; it’s a common issue.

author: HB9EVI
author_rank: Frequent Contributor
timestamp: October 24, 2020, 08:39:35 am
content: don't mistake the MTU (maximum transfer unit) which is 1500bytes for the effective packet size which can always be smaller than the MTU, depending on the payload.very likely the 'problem' is the manner the mobile operator counts the data being transmitted, so even if the payload is only a few bytes every 5min, the operator maybe charges for 5 or 10kbytes minimum each time, that's still common practice for certain data plans.my experience with the W5500 is, that it doesn't send anything what the code in the mcu behind isn't telling it to send.

author: Syntax Error
author_rank: Frequent Contributor
timestamp: October 24, 2020, 09:08:43 am
content: I agree, that's one huge payload but not uncommon for IoT. But every five seconds, suggests a web socket.Two cyber security questions: where exactly is this 'smart' device calling home (a remote server in China) and, how do you control your smart device/s? With the manufacturer's app I assume?Definately 'Shark' your network. Pehaps post the captured packets here (sanitised off course).

author: honeybadger
author_rank: Contributor
timestamp: October 24, 2020, 04:13:17 pm
content: Hi,the device is connected to the client's server. It is not in China. The device is not controlled remotely.Data is somehow repetitive. It doe not matter if one/two "packets" are missing. It sends bunch of temperatures readings, some other readings and short status message.LoRa is not possible. When my client installs it he just needs to plug the Ethernet cable and forget about that.I will paste sanitized Wireshark output next week. Thank you all for suggestions

author: hli
author_rank: Frequent Contributor
timestamp: October 24, 2020, 10:35:32 pm
content: Quote from: honeybadger on October 23, 2020, 09:58:20 pmtcpdump / wireshark will help you. What you might see:each request could do a DNS query (not sure whether the W5500 has a cache for that). These are not freea POST request contains, at least, the hostname of the server, and the URL which the request goes to. These are (together with the header names) easily hundred bytesdepending on how your request looks like exactly, there might be additional headers (a.g. for authentication)and then the server sends a response - which in turn contains at least some headers, and maybe an additional payloadThese all count as traffic. Absent any other infoprmation my guess would that the answer of the server is large.Using WebSockets might help, or just pushing to a MQTT server (which seems to the natural answer to your described problem).

author: amyk
author_rank: Super Contributor
timestamp: October 25, 2020, 01:33:10 am
content: 10GB per month is a constant stream of slightly under 4KB/s, whereas you are expecting closer to 100B/s.Are you actually using HTTPS (HTTP over TLS/SSL)? A single handshake, without sending any application data, can be a few KB for that already.Otherwise, check the fine print of the service provision agreement. Internet service is not truly symmetrical a lot of the time, "2GB data plan" may be referring to download only and upload data is billed at a different (and higher) rate.On the topic of recommending things the OP might not be able to do: use a binary protocol for smallest size and best efficiency.

author: bson
author_rank: Unknown
timestamp: October 25, 2020, 03:23:06 am
content: Since you send something every 5 seconds, make sure the library you use keeps the connection open or you can run into all sorts of other problems as well. More specifically you can run out of ports, or if the library doesn't fully implement TIME_WAIT it can end up trying to use IP address tuples in TIME_WAIT on the server. Then have to try a gazillion ports until it finds one available - and this will be neither reliable or data efficient.

author: girishv
author_rank: Regular Contributor
timestamp: October 25, 2020, 05:34:17 am
content: Quote from: honeybadger on October 24, 2020, 04:13:17 pmUDP is time tested for use cases like this. As you can afford losing few packets here and there, you should consider using UDP.

author: ejeffrey
author_rank: Super Contributor
timestamp: October 25, 2020, 05:36:35 am
content: Quote from: amyk on October 25, 2020, 01:33:10 amI don't think the arduino ethernet shield library supports HTTPs or TLS. The wiznet w5500 certainly doesn't support encryption offload. This seems pretty unlikely. Wireshark packet capture will show for sure...

author: m k
author_rank: Super Contributor
timestamp: October 25, 2020, 08:12:24 pm
content: Quote from: honeybadger on October 23, 2020, 10:32:04 pmRouters have data counters, can you check them?I'm with amyk, your small data plan can be very asymmetric.

author: honeybadger
author_rank: Contributor
timestamp: October 26, 2020, 11:46:47 pm
content: edit:Hi,first off - there is no HTTPS nor TLS.second... Wireshark... interesting.There is something rotten with the "F()" macro!Everything sent with "F()" (for example client.print(F("POST /")); ) is sent char by char.Everything sent without (for example client.print(url_address); ) is sent as a whole.Will test and report.

author: nctnico
author_rank: Super Contributor
timestamp: October 27, 2020, 12:48:41 am
content: Quote from: ejeffrey on October 24, 2020, 03:34:28 amNonsense. UDP is perfect for repeating status updates. Especially if it is a one way status message. All media streaming protocols use UDP because a missing packet is not relevant by the time the receiver finds out it is missing. Especially for realtime communication TCP can be a nightmare. If a packet gets lost there is an incremental time-out period during which there is no communication at all. A few missing TCP packets can jack the time up to 10s of seconds during which there will be no status updates.

author: jbb
author_rank: Super Contributor
timestamp: October 27, 2020, 03:57:41 am
content: Quote from: honeybadger on October 26, 2020, 11:46:47 pmYowza! So much overhead! I suggest you have a look inside the putc function handler (as buried in the F(...) macro).If it’s short, could you try pre-cooking the string with and then sending that whole buffer off in one go?

author: ejeffrey
author_rank: Super Contributor
timestamp: October 27, 2020, 04:34:23 am
content: Quote from: honeybadger on October 26, 2020, 11:46:47 pmI don't remember the details of how it works, but the F macro arranges for string literals to be in program memory instead of RAM, then triggers (via C++ function overloading) an alternate version of string functions that know how to access program memory. I think the typical implementations read a byte into memory at a time, then call the regular print command with a single byte string. If this is done slowly the wiznet chip will not coalesce the writes into fewer packets.The most straighforward fix assuming you don't have enough RAM to ditch all F macros is is a wrapper for client.print() that users strncpy_P() to copy into a temporary variable of a moderate size (say 32 bytes) and then calls client.print() on the buffer. If the fixed string is more than 32 bytes you would still break it up into separate calls but this would reduce overhead dramatically.If you have enough memory available, you could assemble your entire request in a single array and then send it in one go with a single call to client.print(). This would be the most efficient if you have the RAM available but require a bit more restructuring of your code.I think there are ways to do this by mucking about with the wiznet registers. I think you can pause sending, take as long as you want to load data into the wiznet onboard RAM, and then tell it to start sending again. This would also be efficient, and would allow you to avoid filling up the AVR ram as you would be doing the assembly on the wiznet chip. However, this sounds like an enormous pain in the ass to get working properly with the arduino ethernet library.FWIW, I would consider this a pretty big bug in the arduino ethernet library. They really shouldn't be falling back on single byte writes. It doesn't matter for something like a serial port which is what the F() macro is designed for, but it is not really OK for a network interface.

author: Fredderic
author_rank: Regular Contributor
timestamp: October 27, 2020, 07:32:07 am
content: Quote from: honeybadger on October 26, 2020, 11:46:47 pmThe F() macro as pointed out, stores the string in PROGMEM… Problem there — at least on the processors I've used where it's typically a thing — is that PROGMEM is 16-bit, not the usual 8-bit byte addressable. And the F() macro doesn't re-pack the string, so you end up with something like "H\0e\0l\0l\0o\0\0\0" actually being stored — it has no way of knowing exactly what you plan to do with that string, so it sacrifices a little memory for flexibility and ease of access (also, it forms a highly optimisable loop, so it should end up being pretty darned quick). Plus, you tend to have 8x or 16x as much PROGMEM as regular RAM, so it really usually doesn't actually matter for the kinds of tasks Arduino is aimed at (which is not professional high-end development squeezing out every last drop of available resources).But, therefore, you end up with a loop that reads a word (16-bit), writes the lower byte (8-bit), reads the next word, writes the lower byte, etc. Such a loop if it's just doing a memory to memory copy shouldn't actually be any slower once the compiler does it's optimisations, but in the case of writing to a network device … that could well be interpreted by the networking layer as "I want this [1-byte] string content written immediately" (as in, write the character and then do a flush). If that's what's happening, it's probably a case of the Arduino core again, doesn't know what you're doing with that data, so it's just using the standard stream "write a byte" functionality. Which for memory-to-memory is fine, it usually optimises away nice and tight. But if the networking library has a more packet mentality, then it might assume it's being told to send a 1-byte "packet", and goes ahead and does exactly what it's being told to do. I'm really not sure what happens within the networking stuff… This part is mostly just a guess.In any case, that's also why the Arduino typically copies hard-coded strings into RAM at start-up — it does a single mass PROGMEM to RAM copy loop at the beginning, transferring the entire chunk of string space (not necessarily only strings, either) into regular RAM in one go, so it's all nice and contiguous how you'd expect it to be, and other parts of the code using those strings can then just go ahead and use it like usual (pretty sure RAM is generally a little faster, too). Also, packing the strings would require it to know that it even needs to do that (obviously, not all processors do), and have special handling in the compiler (could probably be handled with some super-fancy core library work in modern compilers, but even then what happens if you tell your compiler to use a prior standard — do you expect it to somehow divide up your code between two different compilers?), etc. So it doesn't.Do keep in mind Arduino also really isn't targeted at high-end professional embedded systems developers — don't get me wrong, it's awesome, I love it myself, and most of the time it just works (which is more than I can say for some other compilers…), and doesn't cost thousands of dollars. But when you're trying to squeeze that last bit of space or performance out, it doesn't always make the most helpful choices, and can tend to get in the way… Things like digitalWrite() checking that the pin is in the right mode first…every single time!I really would like a "I know what I'm doing" version…So yeah, perhaps as suggested already, copy the strings into a RAM buffer, and send from there. Unfortunately, sometimes the networking library will then be super helpful by copying that into yet another buffer — especially if the data is then transmitted by an ISR. In one such case, a networking library expected you to give it a fully formed packet, which it copied into it's private buffer leaving room for the header and footer, then after patching them up, proceeded to dump the buffer to the networking device by ISR. The internal function would then block waiting for the last one to finish, before accepting another write — but since you had to build up each packet as a whole first, it was usually well and truly done by then, and would block on you if it wasn't (you could also check if it was done yet).Anyhow, I went in and added a stream-based writer directly into the internal buffer starting at the appropriate location (and limiting it to the appropriate length), and split the "send the packet" function to exclude the buffer copy, with the old function now just doing the buffer copy and then calling new library internal function to patch up the header and footer, and begin the sending. Sounds like more work than it actually was, and it all worked beautifully…But, that was a choice most likely made by the library developer to make it nice and easy for "Arduino type people" to use correctly, and as I say, they explicitly didn't support stream writing specifically to force you to buffer the packet yourself, giving the library time to finish sending the last one in the background, as well as a providing the library with a nice definitive "the data is ready now" signal. In your case, some mixed signals between the Arduino core and the networking librarycouldconceivably be causing individual characters to be interpreted as entire packets… perhaps… In which case buffering the data yourself will indeed fix that — without going to the lengths that I did.

author: Miyuki
author_rank: Frequent Contributor
timestamp: October 27, 2020, 01:48:45 pm
content: W5500 Have a 32kB buffer to easy compose all data into packets, even when MCU working on smaller chunks as Arduino/AVR used here have less RAMSo the only problem here is the library or a bad way of using it

author: cgroen
author_rank: Unknown
timestamp: October 27, 2020, 02:08:09 pm
content: Quote from: ejeffrey on October 24, 2020, 03:34:28 amTotally wrong. UDP is perfect for cases like this. In my work, we send UDP packages from 100.000+ tracking devices over GSM networks all over the world without any problems. On "important data" the servers sends a "ACK" package back to the device (which then removes the message from a circular buffer), the non-important stuff is just sent out. When the device sends a packet (f.ex the "ACK") to a server, the server can return data to the device within several seconds regardless the device is on a "local network" at the GSM providers. Just remember that out-of-order is possible for UDP packets. Everything else is a piece of cake

author: hli
author_rank: Frequent Contributor
timestamp: October 27, 2020, 02:27:52 pm
content: Is the TCP_NODELAY option set on the W5500? If yes, each byte is send immediately. (look up "Nagle's algorithm" for some detail). Disabling it might help.

author: ejeffrey
author_rank: Super Contributor
timestamp: October 27, 2020, 03:01:29 pm
content: Quote from: hli on October 27, 2020, 02:27:52 pmMaybe. I'm not actually sure the W5500 supports Nagles algorithm and it might not do as much as you hope if the code is slow and the network fast. The way I know to do this on the w5500 is to set the port to idle while you write data the set the state to TCP_SEND and let it process the whole chunk. But this functionality might not be exposed in the arduino libraries that prioritize simplicity over performance or functionality.

author: amyk
author_rank: Super Contributor
timestamp: October 28, 2020, 12:20:36 am
content: This whole thread is a great "why abstraction is bad" example. It wastes the resources of the hardware, and of the humans who have to deal with the mess it creates.

author: hans
author_rank: Super Contributor
timestamp: October 28, 2020, 08:44:15 am
content: I disagree. The whole point of writing an abstraction is to check if the output and operation is acceptable. An integration or unit test could have captured that.Otherwise we all better go back to programming Assembly. C is too bloody complex to abstract CPU registers like as if they are variables that move around all by themselves!Using Arduino libraries on the other hand in production... ouch. Finding high quality libraries is far from trivial

author: pardo-bsso
author_rank: Frequent Contributor
timestamp: November 04, 2020, 12:03:41 am
content: Hey... OP,did you solve your issue?

author: honeybadger
author_rank: Contributor
timestamp: November 10, 2020, 10:48:48 pm
content: Hi,quick update, sorry for not replying I don't have time to do anything.client.print(F("Host: ")); - sends every character in separate packetclient.print("Host: "); - sends one packet, BUT each client.print() sends separate packetclient.println("Host: "); - same as previous BUT it sends additional packet for newline charactersOriginal Arduino Ethernet lib is f*cking uselessWill try another lib when I have time.

author: jbb
author_rank: Super Contributor
timestamp: November 10, 2020, 11:50:34 pm
content: If you call client.print(“First\nSecond\n”) does it do that as a single packet?If you’ve got enough RAM you would make a buffer, fill it with snprintf() and then call client.print(buffer)

author: ejeffrey
author_rank: Super Contributor
timestamp: November 11, 2020, 12:46:24 am
content: Quote from: honeybadger on November 10, 2020, 10:48:48 pmThat's definitely a crappy library, but presumably designed for memory efficiency and simplicity above anything else including performance. If you have enough SRAM available just snprintf() into a buffer and client.print() it in a single go, or at least in fairly large chunks. That will be easier than re-building an ethernet library from scratch.

author: Fredderic
author_rank: Regular Contributor
timestamp: November 11, 2020, 04:31:31 pm
content: Quote from: ejeffrey on November 11, 2020, 12:46:24 amThat'sexactlywhat it is. Considering the kind of tasks Arduino is targeted at; even a slow ųC is usually ample fast enough , the Ethernet library already burns a chunk of memory, and asingleEthernet packet itself can be quite a bit larger than thetotal RAM availableon many of the ųC's being used… what size buffer do you expect them to allocate for you, so that you can plod along your merry way and don't have to think about what you're doing? As it is, one of my projects has to use an older version of the standard library because the hundred bytes or so extra bytes used by the newer ones is too much.And further, considering an Arduino generally isn't sending over SSL, you really probably shouldn't be using it anywhere except a LAN anyhow, and those Ethernet chips are likely slow enough you don't have a hope of saturating the link either. So who cares if it sends every character as a separate packet, if it saves having to burn yet more of your super valuable RAM for a buffer?Basically, it's assuming you either know what you're doing, or aren't using it in an environment where sending a single character as a packet is going to be a problem. So, doing it the way it does, leaves the choice up to you:Minimise memory by sending each individual piece of data as a separate packet (no buffering needed).Minimise bandwidth by buffering the data yourself, and sending the whole chunk as a single packet.The thing with F-strings is… annoying. But makes sense; the stream print command isn't buffering, it just takes the data it's given, and sends it as a packet. And again to avoid buffering, the println() command is basically just two print()'s internally, one for the data, and one for the '\n'. You simply need to know these things, and code accordingly.There are also buffer writers right there in the system that will collect your prints for you into a buffer you provide (or use String, which does it on the heap)… This stuff all exists already. It all makes a lot of perfectly good sense when you stop and think it through — though, not that that's ever stopped anything from going horribly wrong. Point is, criticising it before you understand it…Though I'd have thought the networking chip has it's own outbound buffer… but probably also has some tight limits that would be very hard to express to novice users of the library, so they simply opted for one-packet-per-write, and let you buffer it yourself if needed.Personally, I hadn't come across that issue because I almost always prefer the C++ << streaming style into locally allocated buffers — it just feels like a more natural style to me, and compiles down to the same or better anyhow. So I almost always allocate a local buffer, hand it to an instance of a buffer writer, and then when it's all done, just send the completed buffer to wherever it needs to go. Avoids the whole issue that way.

author: ejeffrey
author_rank: Super Contributor
timestamp: November 11, 2020, 09:54:04 pm
content: The chip has an onboard buffer but by default it transmits as soon as it gets data in the buffer. What you can do is pause the socket, execute multiple buffer writes, and then unpause and let it send the entire buffer in a small number of packets. But it is hard to implement that transparently and while it could be exposed to the end user is risky.The other problem is that it is using a simple wrapper to implement the F string version in terms of single character writes. A better library would provide an F string send function that would handle this properly and do so without requiring a large buffer.On the other hand I have a pretty dim view of using any microcontroller for networking much less a dinky AVR. They are just too limited to do it properly even with an offload chip like the wiznet devices.

author: kripton2035
author_rank: Super Contributor
timestamp: November 11, 2020, 10:22:32 pm
content: just curious, does this also happen with the ethernet library of an esp8266 or esp32 using the arduino IDE ?