Dead TCP client detection - socat on a Linux SBC - doing my head in! - Page 1
Forum section: electronics / general

author: Delta
author_rank: Super Contributor
timestamp: December 23, 2017, 01:42:24 am
content: This should be very simple, and probably is for people who know what they are doing, which I clearly don't.I am using a Raspberry Pi 3 running Ubuntu Server 16.04 to receive an ASCII stream on a RS-232 <-> USB converter, and send it to multiple clients over a LAN using raw TCP. (each client connects to a separate TCP port on the server)My script uses socat to read the data from /dev/ttyUSB0 and broadcast it via UDP on the loopback device. (This was the only way I could find that would mean that 1) the data was continually read from /dev/ttyUSB0 even if no further process was wanting it, and 2) multiple processes could read the same stream)Code:socat -d -U udp-sendto:127.255.255.255:29000,broadcast file:/dev/ttyUSB0,nonblockThen multiple instances of socat listen for TCP connections, then read from the UDP broadcast and send it onCode:socat -u -d udp-recv:29000,reuseaddr,rcvbuf=2048 tcp-l:$port,keepalive,keepcnt=5,keepidle=10,keepintvl=2This all works fine, when clients close their connection, my scripts sees that the socat process has stopped and starts it again ready for the next connection. The problem is with clients that do not close their connections properly. This is going to be used in an environment where clients WILL have their network cables unplugged or be hard-powered-down, and the server needs to make the ports available again as quick as possible.For testing, I am using a simpleCode:nc -vv server.ip serverporton a Ubuntu laptop.As it stands, if a client is unplugged, the server keeps the connection open for around 1 hour. (netstat shows the connection as ESTABLISHED)If I add theCode:-T 20to the TCP socats, is will bin the dead connection after 1 minute. (The port then stays in FIN_WAIT1 for a minute, but this is bearable)BUT, the -T option also causes the connection to be closed when the client is still connected but there is no serial data coming in! The keepalives are being sent (when the script and thus the socats) is run as root (I have checked with Wireshark on the laptop), but I assume that as they are only on the TCP link between server and client, socat sees the link between its input (UDP) and output (TCP) as dead, and closes.All connections are unidirectional. Serial into server, TCP stream from server to clients.What I want is:Keep the TCP connection to a client open indefinitely, regardless of whether any data is being sent.Close the connection within 30 secs if there is no ACK response from the client (either from a data packet or keepalive).Re-open ports for future connections within 1 minute.Sorry if this is all over the place, but I hope I have given all the required information.

author: Delta
author_rank: Super Contributor
timestamp: December 23, 2017, 04:20:33 am
content: Well I kind of have it OKish...Code:net.ipv4.tcp_retries1 = 3net.ipv4.tcp_retries2 = 6in /etc/sysctl.conf, and am not using the -T option on socat.This keeps the TCP connection open constantly regardless of serial data flow, and closes the connection to unplugged clients in 30secs.The still annoying thing is if there is no serial data flowing, the TCP connection to an unplugged client is held open until serial data is sent, whereupon it closes immediately.I also have the feeling that it is bad practice to mess around with system settings like this...

author: bson
author_rank: Unknown
timestamp: December 23, 2017, 05:09:35 am
content: Quote from: Delta on December 23, 2017, 01:42:24 amYou should be fine with a single listen process:socat -u -d udp-recv:29000,reuseaddr,rcvbuf=2048 tcp-l:$port,fork,keepalive,keepcnt=5,keepidle=10,keepintvl=2It should never terminate but accept client connections indefinitely, forking a new process for each one. So your script only needs to restart it if for some reason it terminates (like it crashed). A better option might be to run it as a system service, using systemctl (I'm not that familiar rpi's). This way if it dies something is logged and it's automatically restarted. Same with the serial port broadcaster. If you do this they will automatically run whenever the system is booted, so will always be available.

author: Delta
author_rank: Super Contributor
timestamp: December 23, 2017, 06:55:02 am
content: Negative Ghostrider! I tried the fork option, but if multiple clients then connect, the ASCII stream is split between the two clients! Each client receives approximately half of the lines! I gave up with this and just plumped for one socat process and one port number per client.

author: mubes
author_rank: Regular Contributor
timestamp: December 23, 2017, 10:40:27 am
content: Delta,If you can bear going to C then in the _client routine herehttps://github.com/mubes/orbuculum/blob/master/Src/orbuculum.cyou will find code that does pretty much what you are looking for.I'm afraid it's not a zero effort solution cos you will need to remove the depacketiser stuff. Fortunately it's almost only a hacking out job with virtually new code to be added to acheived the result you're looking for.In my experience unixy (bolted together command lines) are fantastic if they do what you need, and _incredibly_ frustrating if they _nearly_ do what you need (or, like me, you just don't have the unix-fu to coax them into compliance).RegardsDave

author: andersm
author_rank: Super Contributor
timestamp: December 23, 2017, 11:43:07 am
content: Have you considered using an existing message broker system (eg. RabbitMQ, Mosquitto) instead of implementing your own?

author: douardda
author_rank: Regular Contributor
timestamp: December 23, 2017, 12:44:18 pm
content: Or the wonderful zeromq library?http://zeromq.orgEnvoyé de mon GT-N7105 en utilisant Tapatalk

author: bd139
author_rank: Super Contributor
timestamp: December 23, 2017, 01:05:40 pm
content: Use zeromq. We send several billion messages a day through thatBeware though, the maintainer died (RIP) so I'm not sure what the future holds for it.

author: shawty
author_rank: Regular Contributor
timestamp: December 23, 2017, 04:01:41 pm
content: Quote from: bd139 on December 23, 2017, 01:05:40 pmI'll back that one up.Failing that, one other option you might want to explore is using C# under mono, which runs really well on the RPi (I use it all the time). The TCP Client class, in the .NET base library (Which is available on the PI under mono) has everything you need.Iv'e not got any code to hand, but what you basically need to do is use the System.Threading Library and The System.Net library. Sit in a tight loop and listen on a single TCP Client object, then as soon as it's connect event fires, spin off a thread, create a new TCP Client, give that new TCP Client the handle from the one you where listening on, and go back into your original listening loop.I wrote an article on this a year or two ago, on the "Dotnet Nuts and Bolts" column for devguru.com , if you go search around on devguru, you'll find it hovering around in there somewhere.My description above doesn't do it justice though, the code was no where near as complex as I'm likely making it sound.If you want to got he C# route, ping me I'll see if I can dig the code out.Shawty

author: bd139
author_rank: Super Contributor
timestamp: December 23, 2017, 04:15:46 pm
content: Just watch out with C#; threads take a lot of memory and kick up the context switch rate. Look at async as well.

author: shawty
author_rank: Regular Contributor
timestamp: December 23, 2017, 11:39:37 pm
content: Quote from: bd139 on December 23, 2017, 04:15:46 pmAgreed.But you could also argue the case against the app not being real-time too. I find it plenty enough for the daemons I use it to write, when controlling stuff.In fact my main concern would be buffering the serial connection at a sustainable rate.I just wrote an output routine last weekend to push data down the serial port at 115200, and sustain a 2Mhz realtime(ish) rate. (I was trying to feed real-time chip programming instructions to a chip wired up to my arduino) [My post on Driving IC clocks in the PIC section of this forum was off of the back of it]

author: grumpydoc
author_rank: Super Contributor
timestamp: December 24, 2017, 08:56:48 am
content: Your architecture for this seems a bit screwy.TCP servers do not normally need to have a separate port for each client, they are supposed to sit on a "well known" port listening for connections. Connections which are not fully closed do sit around for a bit in case the final packets arrive but that is only a problem if you have a high rate of incoming connections and need to free up old ones quickly - it does not sound as though this is a problem in your scenario.The correct way would seem to be to write a small server in the language of your choice, but if you want to stick with command line tools why not use UDP to theLANbroadcast address and pick that up at the client.The client<->server connection is then stateless and clients dropping in and out (be it for unexpected power loss or cables being pulled out) won't affect anything but the client.Do clients need to send anything back to the "server"?