TCP Serial servers - Page 1
Forum section: electronics / general

author: paulca
author_rank: Super Contributor
timestamp: March 12, 2023, 03:40:04 pm
content: Has anyone experience using these devices? I have one in mind for a task. An RS485 device on my network that's near a switch.If am reading it write you basically pair TCP connection and series protocols.On the other end can I assume you can add a second device to return the payload back to serial? Do any Windows based drivers for TCP based VCOM ports exist?Any pros/cons/gotcha's. They don't seem very common in hobby land, most articles and videos are commercials.

author: coppice
author_rank: Super Contributor
timestamp: March 12, 2023, 03:44:45 pm
content: These things are made in huge numbers. I don't know what they get used for in such numbers. They usually follow standards (e.g. RFC2217), and different models generally interwork OK.

author: peter-h
author_rank: Super Contributor
timestamp: March 12, 2023, 04:30:39 pm
content: I am developing a product which has this among many other features. Mine does it transparently; it maps serial port x to tcp port # like 10000.RFC2217 was looked at but was too much work, and very few people actually use it.Here ya gohttps://www.eevblog.com/forum/microcontrollers/sending-serial-data-over-ethernet-which-protocols-are-popular/

author: cantata.tech
author_rank: Regular Contributor
timestamp: March 13, 2023, 05:41:22 am
content: Quote from: paulca on March 12, 2023, 03:40:04 pmSure. I use them. They're usually known as socket to serial bridges.The main thing to keep in mind is how much performance they will need to provide. How many clients etc. If you need big performance then the effort required to develop them increases appropriately with the performance expectation.There's a few basic ones on github as gists. You can implement them in nearly any programming language that you are comfortable with.I like python the best but Java, C# and any other language will do just fine.

author: peter-h
author_rank: Super Contributor
timestamp: March 13, 2023, 08:54:22 am
content: It's usually easy to get raw speed from serial over tcp, but latency can be an issue because most implementations will accumulate serial data before sending out a tcp packet.

author: Gribo
author_rank: Frequent Contributor
timestamp: March 13, 2023, 06:57:53 pm
content: There are also industrial terminal server (Lantronix, Wiznet, Moxa etc), and they have a VCP driver. They are quite expensive though.

author: cantata.tech
author_rank: Regular Contributor
timestamp: March 14, 2023, 04:15:12 am
content: Quote from: peter-h on March 13, 2023, 08:54:22 amKnowing the protocol helps a lot.If you do, then you can inject the appropriate terminal/tcp "breaks" that will mean that packets are sent on their original intended boundaries.This can be done both on serial and tcp.

author: westfw
author_rank: Super Contributor
timestamp: March 14, 2023, 05:42:26 am
content: I used to be pretty intimately familiar with TCP Serial Servers. But I've been retired for a decade, and frankly the target market is a lot different now. The old serial servers were what would be called "very expensive" compared to modern microcontroller serial devices.I don't know if anything beyond "remote terminal access" was ever standardized. As someone said, there is RFC2217, but it's not widely implemented and was only vaguely aimed at "hardware serial tunneling." And then serial connections went out of style ("broadband internet") and all the main vendors and protocol designers lost interest.That means if you want to tunnel some particular rs485 protocol, you'd probably be at the mercy of some proprietary implementation of ... questionable ... capabilities. I do see a bunch of "TCP to rs485 converters" at various price points from vendors from "respectable" to "eBay", but I don't know how they work, or whether they're interoperable. Or whether there are "host side" matching implementations.(And "rs485" is pretty highly ambiguous. You probably need something specifically aimed at a OSI level or two above that.)

author: peter-h
author_rank: Super Contributor
timestamp: March 14, 2023, 09:32:03 am
content: You generally have to buy these in pairs.The product I have does the simplest possible: physical com port 3 = tcp port 10003 (or whatever).The only advantage of using others is that you can get VCP emulators, commercial or free. That's easier than opening a socket and doing r/w to port 10003.There is still a use for running serial over tcp. Most cheaper industrial hardware has RS485 because it is dirt cheap to implement, whereas ETH costs a fair bit more, and is a whole load more code.This stuff was big in the 1980s, with boxes called "terminal servers". They supported say 8 or 16 VDUs. I think they used some sort of tcp to telnet mapping. Good money making those at 1-2kMy then company was doing some stuff like that but IBM coax and twinax compatible.RS485 is just an interface. The protocol will be whatever the product does e.g. Modbus. And there is Modbus over TCP, although if you have low enough latency (much less than the 3.5 character periods) you don't really need it.

author: paulca
author_rank: Super Contributor
timestamp: March 14, 2023, 09:49:04 am
content: The use case, other than the appealing option for an wired ethernet, switched and VLANed UART, was to gain remote access to some solar panel gear in the garage.It all speaks RS485 ModBus and apparently at least the most common registers are standard to some degree. Such that, and I really didn't expect this, different manufacturers displays will read others devices and vice versa. Would be interesting to see if there is a published "standard" set of these registers somewhere.From what I'm reading however it doesn't sound much like it will be "plug and play", connect RS485 port, connect TCP, connect app to VCP driver.I already have an RS485 monitor installed, it's an ESP8266 originally running my code, but now running someone elses more polished version. The downside is, it's "standard monitor" only. You can't modify the config registers.The only way to change a battery charge parameter in the solar charge controller is to go down and connect the laptop to the thing with a USB/RS485 dongle. My laptop has recently expired, screens gone, not worth trying to use anymore. Not hurrying to get a new one. Un wiring it all, bring the controller to the PC and doing it that way is fine for a one off, not for fiddling and testing.So one option is to write a specific Modbus TCP proxy in an ESP32 myself where I control both ends. It means it's Wifi rather than wired but that's fine for my use. This kind of roll-you-own can exclude 90% of the protocol overheads that I don't need and focus on making the EPEver PC app work.... or... just use the registers myself directly with a commandline client.

author: paulca
author_rank: Super Contributor
timestamp: March 14, 2023, 09:52:58 am
content: I'd say this space still has legs. It may no longer be used for long distance industrial and utility network comms, they have all gone GSM etc.Where it still exists in abundance (RS485/Modbus/UART/CAN) is in solar energy tech and battery storage. It has become the defacto interface for interconnecting these.Many Solar PV installers will provide you a contract where by they can see your system, remote in, monitor it, fix issues etc. While these devices may not be exactly the same as a "wired" serial server they aren't a kick in the pants off. Any installation larger than a house might have to span dozens of meters of COM cabling and it's like to be RS485. In an office building however with a wired backbone ethernet these devices could save a LOT on cabling.

author: coppice
author_rank: Super Contributor
timestamp: March 14, 2023, 02:00:52 pm
content: Quote from: paulca on March 14, 2023, 09:52:58 amAbout 10 years ago there were a lot of people in China looking for a cheap ethernet<->serial solution, talking big numbers. Maybe this was specifically associated with growth in the solar industry.

author: paulca
author_rank: Super Contributor
timestamp: March 14, 2023, 02:31:09 pm
content: Quote from: coppice on March 14, 2023, 02:00:52 pmI was watching a documentary on YT about some of the chinese factories and if you think about it's sort of expected, but rather than big white glitzy production lines making a single product in it's millions, often factories pick up whatever work they can and just retool to suit it. That re-tooling involves a lot of room for automation. Being as they are (stereo-typically), they made do. So on the likes of AliExpress you will find a LOT of products which are available in strange abundance and at strange prices.Basically they have teams of people building micro-controller frankenstien systems to augment existing CNC, assembly belts, pick and place, etc. etc.The STM32H7 MCU boards I have been buying from them often come with the most bizarre set of peripherals on the board. Like things which just don't make sense for makers on a platform like that. One example is a load of STM32H7B0 boards which come with a camera and screen, but not a screen you would view images on a tiny oled. I spotted a snap shot of one of these boards on a sellers website sitting in a pic and place machine, where I assume it's job was to identify and orientate a particular custom part feeder.The RS485 boards also seem in abundance and could be related as RS485 buses are very common in factory automation links (I could be wrong).If you want an MCU to take a photo, identify if a part if upside down on a belt and tell a flipper to flip, RS485 over a switched ethernet might be ideal.Other bizarre peripherals include near field comms modules. I'm still wondering what they use those for? To simplify two or more MCUs working together? They also tend to come with a HUGE abundance of QSPI Flash, like sometimes 8Mb or even more. Maybe that is for storing images? Maybe these also get used as assembly line security.

author: coppice
author_rank: Super Contributor
timestamp: March 14, 2023, 03:04:40 pm
content: Quote from: paulca on March 14, 2023, 02:31:09 pmWhat most MCU users don't fully realise about MCUs is 95% of them are developed for one lead application, with a mind to broadening the appeal only if it can be done at trivial cost. They are rarely general purpose in their intent. When you market an MCU and get pull from area you didn't expect that's a massive bonus, and you look like a hero. Boards are similar, even the EVM ones. Remember the MCU was intended for one market. The EVM will be similar. If you are in the application space the MCU or board was designed for you'll look and think its an entirely obvious product.Quote from: paulca on March 14, 2023, 02:31:09 pmNear field comms is huge, especially in China. The volumes on a product like transport smart card readers may be small compared to the actual cards, but its still pretty big.

author: peter-h
author_rank: Super Contributor
timestamp: March 14, 2023, 06:48:37 pm
content: The chinese go for the most obvious low hanging fruit in finished products.Especially if they can rip something off from the West.And they implement it using the easiest way so if e.g. building a consumer router they do it with a linux box and open-wrt (or the present day variant).Running serial over ethernet has been a totally obvious product for decades. When I started my present business in 1991 I looked at distributing a German product which ran 4 RS232 ports over ETH. Non routable; you set up a MAC # on dipswitchesI didn't take it on in the end because the mfg was incredibly arrogant and unwilling to answer straight simple questions with straight answers. The TCP variants came later (the cheap ones).

author: cantata.tech
author_rank: Regular Contributor
timestamp: March 14, 2023, 07:51:10 pm
content: Quote from: paulca on March 12, 2023, 03:40:04 pmHave you started writing it yet? It's a nice hobby project.I would start with writing an emulator on the pc for the device. Are you using Linux, Windows or Mac?Is your development environment PlatformIO, Arduino or something else?

author: nfmax
author_rank: Super Contributor
timestamp: March 14, 2023, 09:11:43 pm
content: Quote from: paulca on March 14, 2023, 09:49:04 amHow odd - I'm currently trying to do something similar myself! In my case I have a Schüco SGI 1500 T plus 2 inverter which has a completely undocumented RS485 interface, but which I suspect is actually MODBUS. I'm using a Devantech ds3484 Ethernet relay module as a data collector: an analogue input for an outside temperature sensor, logic inputs for door closure & locked switches, logic outputs to drive status indicators, and pulse counters for the grid import and solar generation meters. It also has an RS485 port, and if it is configured to communicate using MODBUS TCP, it will forward commands to 'other' modbus addresses over this interface.All this can be configured using the ready built application software that you can download for it, via a web interface. Or you can write your own programs in the Java-like dScript language, in which the application itself is written. All freely available.I've had this module knocking around for years (I bought it on a whim in an end-of-line clearout sale), but it's still available:https://www.robot-electronics.co.uk/products/relay-modules/ethernet-relay/ds3484.html. It, or another model from the same supplier, may be of interest to you.If you do get anywhere spelunking the solar inverter RS485 interface protocol, I'd be grateful if you could share what you discover.

author: cantata.tech
author_rank: Regular Contributor
timestamp: March 15, 2023, 08:08:01 pm
content: Here's a link to something that I used as a base and then modified:https://github.com/GuLinux/python-socket2serial

author: paulca
author_rank: Super Contributor
timestamp: March 22, 2023, 04:16:24 pm
content: Well, this might get bumped up the list of priorities.Long story, in another thread, but my BMS-less 4S LiFePO4 pack is getting out of balance and that is being exasperated by the solar charge controller which appears to have reverted back to normal "lead acid" mode, including the 15V equalisation charge.The problem is... I'm not 100% sure, if I unwire the SCC and take it to the lab for reprogramming over USB that it will retain those settings while powered down for the return trip.Besides, I'd like lower level protocol access than the "monitoring" gizmo I have.The question is... where are the pitfall is creating a bespoke solution just for the specific needs of Modbus over TCP?I have a dev board here which includes a F411 an ESP32, a screen and a UART header. I also might possibly have a logic level converter board and maybe even a MAX4855 converter board in a parts bin somewhere. That should take care of the hardware, if I can find that stuff.For software, the STM32 can handle the UART/RS485 side of things and just feed the ESP32-AT firmware buffers. The ESP32-AT firmware supports several options on what is usually referred to as "TCP_NO_DELAY". It's how long to wait before sending an incomplete TCP segment. By default the TCP stack will wait for a period of time before sending the partial segment. This is often governed by some fairly non-trivial algorythm, but for me, all I need to care about is any particular protocol timing requirements are met and the "responsiveness" is appropriate for the job.Thus, I can tell the ESP32 TCP stack to buffer the data I send it and timeout after X miliseconds. Thus capping the "packetisation" delay at X milliseconds.Beyond that it should basically be a data agnostic bidirectional buffer.There are always going to be "But what about...."

author: coppice
author_rank: Super Contributor
timestamp: March 22, 2023, 04:49:32 pm
content: Quote from: paulca on March 22, 2023, 04:16:24 pmThat is a standard part of a TCP implementation, turned on by default. Its a gotcha for many beginners who can't understand the high latency they get with their TCP communications code. It is usually referred to as the Nagle feature or algorithm, after the guy who introduced it -https://en.wikipedia.org/wiki/Nagle%27s_algorithm

author: paulca
author_rank: Super Contributor
timestamp: March 22, 2023, 05:08:33 pm
content: Quote from: coppice on March 22, 2023, 04:49:32 pmThat's the name I was looking for. Obviously it lowers your overall through put.There are similar complexities around acks, selective and early versus late and contiguous and all manor in between.It's one of those topics I studied to the n'th degree in Uni. I even did an exam question on the TCP sliding window algorhtm and calculating the maximum through put on a connect with a given latency and a given window size.I do recall using this in anger in work when resolving low latency issues in the stock exchange and it was the customer who reported the Nagle aglo did not appear to be running on "some" of their sessions.

author: coppice
author_rank: Super Contributor
timestamp: March 22, 2023, 05:16:11 pm
content: Quote from: paulca on March 22, 2023, 05:08:33 pmWhy would it affect your throughput? As the data flow builds up Nagle stops doing anything. The algorithm reduces the flow of inefficient small packets at low data rates, but in exchange it creates latency when you are waiting for the algorithm to kick out a packet on timeout.

author: paulca
author_rank: Super Contributor
timestamp: March 22, 2023, 05:28:33 pm
content: Quote from: coppice on March 22, 2023, 05:16:11 pmSorry I was thinking about the result of shortening the time too much, rather than Nagles which tries to balance things.

author: coppice
author_rank: Super Contributor
timestamp: March 22, 2023, 05:41:11 pm
content: Quote from: paulca on March 22, 2023, 05:28:33 pmI've always felt like they missed an element in these TCP implementations. A feature that lets you have Nagle running, but kick out the data in the buffer immediately. I've done a number of things where Nagle is useful, but there are points where I need the buffer to flush to the far end, so it can promptly construct and send its response to me. I have to turn Nagle off, put the piecemeal parts of my request together in a buffer, and send it in one go. If I'm writing all the code that's usually no big deal, but it can be clumsy when working with some libraries.

author: paulca
author_rank: Super Contributor
timestamp: March 22, 2023, 05:55:02 pm
content: There are parts of TCP/IP stack are look unfinished, or at least never reached proper standardisation.Examples are the use of "PUSH" flags on TCP segments. Nobody agrees on what they should do and most routers remove them.The idea around the PUSH flag is "Make the application aware of this segment immediately." Rather than just carrying on filling your network stack buffer before waking your process when it feels like it, but to do it NOW.Nagles has quite a few tuning parameters at the C level and there are even swap in algorithms in the Linux Kernel if you are in that space. Sometimes it just turns into a whole load of testing and measuring and effort to gain 0.15%.

author: coppice
author_rank: Super Contributor
timestamp: March 22, 2023, 06:24:15 pm
content: Quote from: paulca on March 22, 2023, 05:55:02 pmSupport for expedited data is there, but pretty weak.Quote from: paulca on March 22, 2023, 05:55:02 pmThe effect can be really big with client server type applications.Looking at what is happening with HTTP3, I guess SCTP now has no chance to mature and become the flexible transport of choice. Things like firewalls hindered the development of streaming quite a lot, but now things like STUN have matured, and the firewalls hinder anything else that's new, it looks like NATed UDP is the way everything will go. The old firewall joke was firewalls are not a problem, do all comms on port 80. Now it seems to be firewalls are not a problem, make everything look like streaming media.

author: paulca
author_rank: Super Contributor
timestamp: March 22, 2023, 08:45:26 pm
content: Its the fact that Qik w/ HTTP3 can tunnel many connections within the same stream and within the end to end encryption. That scares firewall folks. So much so a lot of places disable Qik and the browsers call back to HTTP2 style single connections again.

author: coppice
author_rank: Super Contributor
timestamp: March 22, 2023, 08:50:30 pm
content: Quote from: paulca on March 22, 2023, 08:45:26 pmInteresting. I thought Qik was succeeding in going where no SCTP connection has managed. Maybe we will still see wide usage of SCTP.

author: westfw
author_rank: Super Contributor
timestamp: March 23, 2023, 12:05:09 am
content: QuoteNagle does "kick out" the first packet immediately; it's subsequent packets that have a delay.There are "interesting" interactions with "delayed ack." :-(Nagel and Delay Ack were both algorithms aimed at the inherent inefficiency of small-packet traffic (ie telnet), back when that was a major application on the net. 160+ bytes sent on the network for each character typed and echoed was ... undesirable.By now, no one much cares about interactive small-packet traffic :-(And I figure that the "slow start" and other modern "congestion-avoidance" algorithms pretty much devolve to Nagle-like behavior when you're not sending full packets, which is fine for bulk data, but not so much for the small-packet traffic.I know of at least one TCP implementation that implemented a limit on the number of TCP packets in the retransmission queue (5 packets, IIRC.) This had an effect similar to Nagle, only ... later. I'd be a bit surprised if microcontroller TCP implementations don't do something similar. (OTOH, the retransmission queue is theoretically bytes rather than packets, and maybe modern implementation are better at treating it that way. I have not kept track, and I'm afraid to do so :-( )QuotePSH and URG are pretty ambiguous (a bit of change of philosophy in how the OS communicates with the network layers, I guess), but I don't think that anyrouterswill change that level of of a packet...(Serial Servers, OTOH, are likely to ignore and either never set or always set PSH.) (URG was used by the "flush output" feature of TELNET. Usually incorrectly and/or to little effect.

author: peter-h
author_rank: Super Contributor
timestamp: March 23, 2023, 08:18:44 am
content: When I was finishing off a serial over tcp feature (code written by someone else) I found I got it to work much better if I disabled Nagle.Nagle may be useful running stuff over mobile data (GPRS, 3G 4G) where latencies can be measured in seconds.QuoteThat's funny. The more modern version would be port 443I travel a fair amount and find that lots of wifi networks block various ports e.g. smtp, pop, etc but 443 always works. A colleague helped out and set up the Softether VPN terminating back at home.Lots of networks block anything to do with video (a good way to trash the data allowance of an apartment you are renting is to upload a few GB to YT or Vimeo) or VOIP.Not sure how well (back on the topic) traffic on port 10001 etc will get through, but nothing stops you using port 80 or 443I've just tried running data over port 80, which didn't work (probably because there are so many things listening on port 80, like browsers) but 443 works fine.

author: paulca
author_rank: Super Contributor
timestamp: March 23, 2023, 09:44:58 am
content: You won't get 443 or port 80 out of my work desktop. No sir. "ping google.co.uk": Host not found. Ping 8.8.8.8 Tmeout.No external DNS even. The ONLY thing you have access to is the corporate proxy, which runs way, way too much high security stuff for you to sneaky a PPP over HTTPS tunnel for example. I haven't tried as I like my job. However, the inability to install software yourself and the total lack of any admin rights anywhere kinda make it not worth trying.

author: ejeffrey
author_rank: Super Contributor
timestamp: March 23, 2023, 03:33:32 pm
content: Quote from: coppice on March 22, 2023, 05:41:11 pmLinux at least has TCP_CORK which completely prevents partial frames from being sent, and then let's you uncork when you need to flush. This is used in e.g. web servers that need to send some headers and then a file.Nagle is really intended for very simple protocols, such a character and line based interactive sessions. It was also designed for an environment when congestion was a much bigger problem than it is today like half duplex Ethernet with collisions. It's sort of expected that if your application is both complex enough and performance sensitive enough that Nagle is a problem you will take over active management of the transmitting. Building up an entire request in an io vector and then sending it with a single system call is basically what your are supposed to do. The main motivation for TCP_CORK was to work alongside sendfile in http servers which avoids loading the payload into user memory entirely.

author: Mark0x01
author_rank: Newbie
timestamp: March 24, 2023, 01:32:20 am
content: A modbus tcp gateway will do the job, so you get modbus over tcp to modbus over serial.No issues with packeting up the data, since it is already done.My Homeassistand uses one with a number of devices on the RS485 cable to monitor mains power, temperaturea and drive the irrigation system with RS485 modbus relay boards.Mine was from Aliexpress, or you can easily find code for esp32 or arduino to do the same thing - I have an esp32 serial modbus client I build from modules.

author: peter-h
author_rank: Super Contributor
timestamp: March 24, 2023, 06:38:12 am
content: QuoteIf these work for general data, they must be despatching the packets purely on a timeout.The one I have is configurable, from a config file setting. I spent a lot of time playing with that... If you want a latency of a few ms or less then you can't do that though. And 3.5 characters at 9600 baud is 3.5ms.Also you can have an instrument (a modbus slave) with a large number of registers and when the master does a block read, the "packet" can be a few k, so you need a buffer big enough for that.I think a general converter will work for modbus in most cases, because apparently the 3.5 character packet delimiter is unimplemented.Most modbus applications are master/slave, where the master polls and the addressed slave responds. So the master's poll will always be conveniently terminated by, ahem, no more dataSo if your modbus over tcp implementation just uses say a 100ms timeout to detect end of packet, it will probably "just work", and I bet this is what the chinese do.The other way is to get involved in the modbus protocol, perhaps speculatively trying out if the last 2 bytes you have is a valid CRC... AIUI the proper modbus over tcp protocol sets up a layer underneath modbus, so whole packets are identified, packaged (potentially over multiple MTUs) and unpacked at the far end. But then such a product isn't going to work for normal data.How else can one do it?

author: paulca
author_rank: Super Contributor
timestamp: March 28, 2023, 06:28:46 pm
content: Got one today. Hooked it up, struggled a little but got it working 90% anyway.The VCOM driver works. The solar controller software connects to the VCOM. Most of the config and monitoring appears to work, only 1 or 2 less important dialogs produce "Operation failed" reading the data.I did make a mistake though. I read the one I bought as supporting RS485. Oops. No, I bought the RS485 model. It has the RS485 to UART converter inside.That's great, but it snubs my use as a generic network UART. They are only £30. I can get another later for that use case.I figure this at least has legs to run with. I'll check out how it behaves on a bus with other stuff as I think I'll end up with 2 or 3 RS485 devices on the bus. I do know there is limited flow control and colisions are bound to happen, but nothing is "transactional".