Serial port over internet, peer-to-peer and through both firewalls - Page 1
Forum section: electronics / projects

author: voltsandjolts
author_rank: Unknown
timestamp: October 05, 2016, 02:15:40 pm
content: I would like to connect to a RS232 serial device on the other side of the world.I would also like to do this through both corporate firewalls without changes to either of them.Ideally, I would like to send a client an 'intelligent adapter' which they plugin to our product serial port and to their network (lets assume internet access allowed without logon/authentication, DHCP sets up gateway address etc.). A similar adapter or virtual comm port is used at the other end. No firewall changes (opening ports) at either end.So, I am thinking, "UDP hole punching with rendezvous server" kind of setup. Similar to, say, VOIP or Dropbox but with serial ports at the end.A pure software solution might behttp://www.flexihub.com/Are there any other options out there?

author: StuUK
author_rank: Frequent Contributor
timestamp: October 05, 2016, 02:17:47 pm
content: http://www.taltech.com/tcpcomhttp://www.eltima.com/rs232-over-ethernet/maybe?

author: voltsandjolts
author_rank: Unknown
timestamp: October 05, 2016, 02:42:15 pm
content: Quote from: StuUK on October 05, 2016, 02:17:47 pmMaybe, but I doubt it. They don't give much technical explanation nor do they mention firewalls at all.They do talk about connection over internet but that is probably achieved by opening ports.I'll download trial versions if nothing else turns up.

author: StuUK
author_rank: Frequent Contributor
timestamp: October 05, 2016, 02:44:45 pm
content: The assumption would be that firewalls are configured appropriately for the ports/addresses that are trying to communicate which (if over the internet) would be down to the local and remote end firewall configurations.

author: voltsandjolts
author_rank: Unknown
timestamp: October 05, 2016, 02:47:32 pm
content: Quote from: StuUK on October 05, 2016, 02:44:45 pmThanks for your input but you didn't read as far as the second line in my original post:"I would also like to do this through both corporate firewalls without changes to either of them."

author: StuUK
author_rank: Frequent Contributor
timestamp: October 05, 2016, 02:51:59 pm
content: I did which is why I mentioned the assumption that the firewalls are configured appropriately, i.e. fairly relaxed on source/destination addresses and ports.Any solution you look at will inevitably require the local and remote ends to be configured with a port to use as a minimum. If you can utilise a port already open and the source/destination addresses are ok then you're good to go.If the corporate firewalls are pretty locked down then it'll be an interesting ride....

author: voltsandjolts
author_rank: Unknown
timestamp: October 05, 2016, 03:13:02 pm
content: Peer-to-peer communications do not necessarily require open IP ports (as many popular applications demonstrate).I am specifically looking for a solution which does not require open ports.Thanks.

author: StuUK
author_rank: Frequent Contributor
timestamp: October 05, 2016, 03:31:18 pm
content: Popular 'Peer to Peer' applications generally use a signalling server to negotiate connection details. I'm assuming that you require something that you 'plug' in both ends and communicate directly, presumably over the internet, and don't involve a third party service/server but I can see I'm not helping so i'll duck out.

author: Emo
author_rank: Regular Contributor
timestamp: October 05, 2016, 03:45:51 pm
content: Hi,I think the virtual solution from eterlogic might be a solutionhttp://eterlogic.com/Products.VSPE.htmlI use it for several years now on linux and windows.regards,Eric

author: mariush
author_rank: Super Contributor
timestamp: October 05, 2016, 03:45:59 pm
content: If the transfer rates are relatively small you could make your product completely passive as in don't allow anything to connect to device, let the device send data to a web server and request incoming data from same web server periodically.Give each adapter a unique Id ( which can me stored in the microcontroller's eeprom memory or something like that) and the micro connects to the website and does a request that looks something like this GET /receive?uid=12345&lastpacket=24354 wherethe uid is the unique id of the device and lastpacket is the unique id or time stamp of the last packet of data the server sent to you, so that in case connection is suddenly broken , the adapter can reconnect and resume transfer of data without losing anything.in theory the connection can be kept open up to minutes or hours, because that's how online radios typically work these days, so ISPs shouldn't interrupt you. But in the worst case that you have a bad router or switch that kills connections or they freeze after some time, the micro can close connection and create another request to the remote server using the last packet id as resume point.As for sending data to user, this could also be done using a simple POST http request to the web server with data encoded in a variable or optionally declaring the data as a file upload of unknown/very long size and just killing the connection when you feel like stopping transferring messages. you can use even a GET request encoding the data to send in a parameter in the URL, but you'd have to be careful about length of URLs (less than about 4K would probably be safe)On the server side, the incoming and outgoing data could be kept in RAM using a memory / disk backed database (think memcached but with no loss guarantee, keep data in ram and delete packets as soon as they're transferred and no longer needed to be stored)The problem with this system is the added latency between device and user - you get the latency between device and server and then from server to user and matching but otherwise with html 5 and javascript these days you can make some kid of terminal fairly easily. Also, if the user turns out to have a static IP or that your application can break through his firewall and your server can connect from outside to the user's computer inside a network, it could optionally instruct your adapter to talk directly to that user's ip:port pair until disconnected, and then fall back to default device-server-user behavior.Think o all "shoutboxes" that are on various forums which pretty much work on the same concepts.ps and html 5 also has web sockets which makes chat back and forth very easy.ps. my keyboard died, using a temporary shitty one with very hard to press buttons so in my posts i may miss some letters, sorry if they look bad or are hard to read

author: voltsandjolts
author_rank: Unknown
timestamp: October 05, 2016, 06:16:38 pm
content: I was looking for an off-the-shelf solution but maybe it could be a back-burner project.I was hoping for peer to peer (using initial 3rd party/rendezvous server) just to keep latency down but it seems P2P is not always possible behind firewalls - depends on the setup:http://stackoverflow.com/questions/23176800/whats-so-hard-about-p2p-hole-punchingPassing all data through a cloud webserver might be the more reliable way to go, albeit a little slower. I guess there are many ways to tackle that but websockets would seem the modern way, perhaps a raspberry pi at each end using python pyserial/websocket to cloud server which would connect the two sockets together.Latency on this websocket server seems OK:https://www.websocket.org/echo.html

author: voltsandjolts
author_rank: Unknown
timestamp: October 05, 2016, 09:12:59 pm
content: I'll givewww.flexihub.coma try.Note to future self:I think a DIY solution could made out of python +http://autobahn.ws/python/+http://crossbar.io/Example serial port over websockets here:http://tavendo.com/blog/post/arduino-yun-with-autobahn/

author: IanJ
author_rank: Unknown
timestamp: October 06, 2016, 10:43:13 am
content: Hi,Been many years, but I am sure I used Lantronix serial servers to do this, and configured them to use port 80. Tested city to city and through Cisco routers/firewalls.Devices were Lantronix MSS1 & MSS4 but I am sure there will be new models now!Ian.

author: voltsandjolts
author_rank: Unknown
timestamp: October 06, 2016, 11:47:22 am
content: Thanks Ian but it looks like all their current serial servers require ports to be opened for internet comms.

author: Jeroen3
author_rank: Super Contributor
timestamp: October 06, 2016, 01:44:04 pm
content: You can try aneWon. However, it looks like you need two m2m and not h2m.Theoretically it should be possible to connect two eWon sites together. But you'll have to ask them.You get an OpenVPN network in their cloud to which you can connect with your laptop. The uplink device in the machine also connects to this cloud. Not firewall changes required, except vpn pass-trough I think.Quote from: voltsandjolts on October 05, 2016, 02:15:40 pmYou have two problems with that:1. How do the devices know each others WAN IP.2. A firewall surely does not accept random UDP frames except from the destination of the outgoing UDP frame. Or does it?It's what UPnP was invented for, right?

author: Aeternam
author_rank: Unknown
timestamp: October 06, 2016, 01:46:20 pm
content: Quote from: voltsandjolts on October 06, 2016, 11:47:22 amPorts are an integral part of the communication infrastructure of a computer network, there's no way around them.https://en.wikipedia.org/wiki/Port_%28computer_networking%29There is no (easy and/or legal) way you will be able to communicate with a device or service behind a firewall if the firewall is configured to block the port through which you want to address your device. HTTP (the protocol used for web browsing) for instance uses port 80. Now usually port 80 is open on the corporate firewall for people to be able to check their webmail or shop online and youcoulduse it to communicate with your device. But if the firewall blocks that port, you can't surf the web and you won't be able to reach your gear.In your opening post you make the assumption of an "open internet access and default gateway config." If this is the case (on both ends) you will be able to reach both of your devices through port 80. You would still need to know their public IP address in order to connect to them.Be advised also that network admins (especially the ones in charge of large corporate networks) do not like traffic on their network that they don't know about. You might want to talk to them before hacking away

author: Jeroen3
author_rank: Super Contributor
timestamp: October 06, 2016, 01:58:10 pm
content: There is a fundamental difference between source port 80 and destination port 80.To view a website,youuse destination port 80 on. To do that you use source port anywhere from 49152–65535.Blocking port 80 will not prevent anyuserfrom accesing webmail or webpages over http.Firewalls usually do not care about destination ports. Unless they are specific to some protocol. Such as torrents.

author: nuno
author_rank: Frequent Contributor
timestamp: October 06, 2016, 02:23:56 pm
content: You'll need a server between both and outside both local networks, where all data has to go through (not only for initial connection), I see no way around it. Maybe a PHP/.NET/servlet/whatever web page could be used thus avoiding the more onerous way of having a full server under control. You might get away with some free IM way, not sure what's available today.It would be a lot easier if your system could accept a connection on one side (such as on your company's side).

author: voltsandjolts
author_rank: Unknown
timestamp: October 06, 2016, 02:36:10 pm
content: @AeternamAs Jeroen3 says, you are misunderstanding how ports work.Quote from: Jeroen3 on October 06, 2016, 01:44:04 pmThanks, I'll have a look at that.QuoteThat's the job of the rendezvous server, to initiate the connection, tell each end the IP address of the other.QuoteThat's what UDP hole punching is for but, unfortunately, some networks may disallow it:http://stackoverflow.com/questions/23176800/whats-so-hard-about-p2p-hole-punchingUPnP won't work in corporate network.

author: voltsandjolts
author_rank: Unknown
timestamp: October 06, 2016, 03:05:42 pm
content: @Jeroen3Ooooo, this might be the jackpot. It even says Firewall Friendly! Now to test if I can get OpenVPN outbound from here....

author: CaptCrash
author_rank: Regular Contributor
timestamp: October 06, 2016, 03:23:51 pm
content: Quote from: Jeroen3 on October 06, 2016, 01:58:10 pmThe source port range depends on the client software, recent versions of windows use the ranges you have listed, older versions use 1025-5000, other operating systems/IP implementations will use other ranges of source ports.https://support.microsoft.com/en-us/kb/929851]. To do that you use source port anywhere from 49152–65535.The source port range depends on the client software, recent versions of windows use the ranges you have listed, older versions use 1025-5000, other operating systems/IP implementations will use other ranges of source ports.https://support.microsoft.com/en-us/kb/929851Quote from: Jeroen3 on October 06, 2016, 01:58:10 pmIf you mean, blocking a source port of TCP 80, then yes you are correct.If you mean, blocking a destination port of TCP 80, then no you are incorrect.e.g. If you add a rule that blocks or drops all traffic with a destination of TCP port 80 and place this rule at the top of the rules list (giving it the highest priority), then you will not be able to browse to any external HTTP (TCP port 80) traffic via this firewall.Quote from: Jeroen3 on October 06, 2016, 01:58:10 pmFor outbound traffic, Firewalls only care about destination ports (you can also limit source ports but its not typical). What you have written is entirely backwards.Quote from: Jeroen3 on October 06, 2016, 01:58:10 pmThere is nothing different about one protocol (port) from another from a firewalls perspective, connections are allowed or not (there are some special cases, like established sessions, responses from other ports like with FTP, but generally its fairly straight forward).Torrents are in no way a special case, they have the exact same limitations/capabilities as any other protocol in that they need to be allowed through the firewall.This requires either explicit rules on the firewall (or very broad rules that just allow everything out) or via some automatic process like UPnP).

author: CaptCrash
author_rank: Regular Contributor
timestamp: October 06, 2016, 03:27:12 pm
content: Quote from: voltsandjolts on October 06, 2016, 02:36:10 pmSorry but Jeroem3 has it backwards.@Aeternam got it correct in terms of how ports work and how any reasonable network admin would approach configuring a firewall.

author: voltsandjolts
author_rank: Unknown
timestamp: October 06, 2016, 03:42:05 pm
content: Aeternam is misunderstanding the term "open port", or at least what he has written suggests that.

author: David Hess
author_rank: Super Contributor
timestamp: October 06, 2016, 09:33:13 pm
content: Assuming that the firewalls allow outgoing UDP, I would look into using preconfigured OpenVPN network appliances on both ends which automatically connect to a central OpenVPN server so the network endpoints then see each other as local. TCP will will work for this as well although probably not quite as well and if desperate and port 80 is not through a proxy, outgoing port 80 could also work.It has been a while since I have used OpenVPN but I think this is possible to do this without knowing the endpoint IPs. The endpoints just need to know the central server's IP.

author: wine+dine
author_rank: Contributor
timestamp: October 07, 2016, 07:06:50 am
content: There are many possible ways in which firewalls are configured. If you have to work with a corporate firewall on either side that you don't know about in detail and can't have modified, this can be a hard problem.A simple firewall in the past may have just filtered connections based on addresses and ports, e.g. "from internal to anywhere with destination port 80 except those on the blacklist". In such a case, a reflector server can listen on port 80, accept connections from both endpoints, identify them and relay data from one to another."Next-generation" firewalls such as the one I manage for my employer perform deep inspection of the traffic and know about the lowest level details of application-layer protocols, not only network-layer. What that means is that the port-80 reflector traffic would be blocked even if connections to web servers on port 80 are allowed, because your data would not look like legitimate HTTP.Vendors such as Palo Alto have turned this sort of inspection into an art form, with decoder logic updating every few days to match developments, and rule granularity available down to allowing specific functions on a service but not others, e.g. Gmail yes but attachments no. (The decoding is done at gigabit+ speed on FPGA-based hardware, BTW.)The web sockets idea is a good one but keep in mind that the above means there is no absolute guarantee it will work even if both sides can connect to your server.

author: voltsandjolts
author_rank: Unknown
timestamp: October 07, 2016, 08:09:34 am
content: Quote from: David Hess on October 06, 2016, 09:33:13 pmThanks David.This is what the eWon (https://ewon.biz/) suggested by Jeroem3 does - they provide the central cloud server. I'm looking into this now.FlexiHubFor the record, thewww.flexihub.comsolution I mentioned earlier will let you connect to remote devices over the Internet without the need to configure port forwarding on your router (i.e. opening ports for new inbound connections). However, it needs to use destination ports which are blocked on my corporate LAN so I couldn't even login. For a less restrictive corporate LAN it may actually work.

author: Jeroen3
author_rank: Super Contributor
timestamp: October 07, 2016, 09:29:20 am
content: Quote from: CaptCrash on October 06, 2016, 03:27:12 pmStandard firewalls care don't about outgoing traffic that much. Sometimes they can block a specific outgoing protocol. (based on dstport) Blocking dstport 80 is something I have never seen before. Most of the times a proxy is installed with filtering if blocking web content is the goal. Or even worse, a filtered DNS. It's an especially outdated method since a lot of websites use dstport 443 today.My reply was based on the common misconception on ISP-forums that people need to port-forward tcp80 to their pc/xbox for "internet" to work.Not knowing the knowledge of voltsandjolts, we could be sending him to the network guy asking for something similar to blinker fluid.Quote from: voltsandjolts on October 06, 2016, 03:05:42 pmIndeed. We use it to plug into any random network our mobile analyzer is temporarily mounted. If there is no network, we use cellular. And since cellular used carrier grade NAT, you can't use server concepts on this side of the link.OpenVPN is rarely blocked since you'll need at least an medium-bearded-IT-guy to be experienced enough that he blocks it. Those are rare.

author: David Hess
author_rank: Super Contributor
timestamp: October 07, 2016, 02:27:38 pm
content: Quote from: voltsandjolts on October 07, 2016, 08:09:34 amThe server part could be setup by you at any fixed location or on a cloud server so you would retain complete control. Technically oriented people with an interest in having their own VPN endpoint for security often do this instead of buying a subscription to a VPN service so the instructions for using OpenVPN this way are out there.

author: voltsandjolts
author_rank: Unknown
timestamp: October 07, 2016, 03:08:06 pm
content: Yes, I like the idea but its probably a bit over my head to be honest.With a virtual private network setup I could then use com0com or something similar to share a comport over that virtual network connection.I see Amazon do hosted OpenVPN Access Server as a cloud servicehttps://openvpn.net/index.php/access-server/cloudmachines.htmlbut I'm reluctant to dive in.LogMeIn Hamachihttps://www.vpn.net/makes small virtual networks free and easy (peer to peer too for reduced latency) but, alas, blocked on my corp network.Also, SSH tunnel between computers on corp networks might be possible withhttps://openport.io/but I haven't tried it.It seems an OpenVPN solution is the best way forward for me and I'll probably go with eWon.Thanks for your input.

author: Jeroen3
author_rank: Super Contributor
timestamp: October 07, 2016, 04:11:43 pm
content: There is also the cost factor. Although it seems cheaper to set up such thing yourself, it isn't.- You need to buy local hardware.- You need to rent a VPS- You need to setup and maintain (eg: firmware updates) the local hardware.- You need to setup and maintain the vps.- You need to be able to fix downtime 24/7, and have knowledge to do this.

author: Kalvin
author_rank: Super Contributor
timestamp: October 07, 2016, 04:36:32 pm
content: - Raspberry Pi- Serial-to-USB-converter- SSH server- Picocomm or similar applicationIf you can create an SSH tunnel from your machine to the Raspebrry Pi, your are done: Just SSH into the Raspberry Pi and launch the serial terminal program. If the firewalls block all incoming traffic, you would need a proxy-server (Possibly another Raspberry Pi or similar gadget connected to the public Internet). Anyway, it is doable.https://toic.org/blog/2009/reverse-ssh-port-forwarding/