web, how to redirect a serial port to a web page? - Page 1
Forum section: electronics / general

author: DiTBho
author_rank: Super Contributor
timestamp: December 06, 2022, 04:15:21 pm
content: I'd like to allow people to use my old computer boards (68k, 68hc11, etc), but I cannot open nothing but a port 80(1), so I'd like to write an application (PHP?) thatinterfaces GNU/Linux /dev/ttyS1redirects input/output to a web pageuses a "target reset" button to launch a tool I have made to issue a physical resetallows the user to download S19 binary, which will be then passed to a proper toolimplements a semaphore: if the target is busy with a user, other users should waitafter T-minutes of inactivity, the semaphore should be g greenis it possible with PHP?I have ZERO know/how in this kind of web applications(1) must be "HTTP", you can't ssh-tunnel, otherwise the firewall will start kicking like an angry donkey.

author: ledtester
author_rank: Super Contributor
timestamp: December 06, 2022, 04:36:45 pm
content: This app:https://github.com/serialport/node-serialportsets up a (node-based) HTTP server which gives access to serial ports. You can then use javascript in the browser to read/write to serial devices.More info athttps://serialport.io

author: Nominal Animal
author_rank: Super Contributor
timestamp: December 08, 2022, 08:27:35 am
content: Quote from: DiTBho on December 06, 2022, 04:15:21 pmHave you checked if the firewall allowsWebSockets? Theystartas HTTP, but use a HTTP/1.1 upgrade header to switch to WebSockets. It doesn't work over proxies, unless the proxy explicitly supports WebSockets.For a pure HTTP (non-WebSockets) solution, you need a serial port management agent, running concurrently but separately from the HTTP server. This is basically a small buffering daemon that talks to the serial port. The web pages use a CGI program or script – PHP working fine for this – that talks to the buffering daemon.It generates quite a bit of HTTP traffic, since any new input or output requires a new request, so on a small box it can be quite a significant load.If your firewall passes WebSockets connections, and you can install a websockets forwarding module in your web server (Nginx has it built-in since 1.3.13; Apache needsmod_proxy_wstunnel, or in 2.4.47 and later,mod_proxy_http), then you have much more options.You still need a management agent running on the server, talking WebSockets protocol. The web server acts like a WebSockets proxy, connecting to the port (for example, in the loopback interface, say 127.0.0.1:8080) the agent listens on.If the other end is a web page talking WebSockets over JavaScript, the datastream needs to consist of text or XML, so that it is easily parsed and constructed in JavaScript.However, you can also create a WebSockets client program, that creates a pseudoterminal pair, with the slave end representing the remote serial port. That way, the serial port can be accessed much like a native port, remotely, over the HTTP/WebSockets connection on port 80. (The only difference is that without nonstandard kernel or linker support, we cannot really tellwhentermios settings are changed, as they need to be polled: we cannot tell whether termios settings were changedbeforeorafterthe end client did a read or a write.)On Linux, a simple interposing library can be implemented to avoid the issue, with or without a pseudoterminal pair. The WebSockets client program runs whatever command specified, typically shell, but with an interposing library loaded viaLD_PRELOAD_LIB. This library interposes open(), close(), read(), write(), tcgetattr()/tcsetattr()/tcsendbreak()/tcdrain()/tcflush()/tcflow(), ioctl() (for tty_ioctl's) etc., and if they refer to the pseudo-file or socket representing the serial port, they are emulated via a socketpair to the client program which forwards everything to the agent on the server via WebSockets.(The pseudoterminal pair would make things simpler, since only open()/openat(), close(), ioctl(), and the termios-related functions would need to be interposed, and data flow "naturally" through the pseudoterminal pair.)In essence, any normal Linux program accessing the "pseudoport" would act the same way if run on the server with the hardware serial port, except with considerable latency per interposed syscall (typically in the millisecond range, due to the internet round-trip time).Since this is HTTP, security is going to be limited, but you can definitely add challenge-response access control (via e.g. username and password), with only a dual-salted password ever transferred, preferably with a time-dependent part embedded in the salt, done during the initial connection.If this would help with your work among your team and friends, I think I could sketch something more concrete; I'm just quite rusty, so I'd need to recheck many details first.

author: DiTBho
author_rank: Super Contributor
timestamp: December 09, 2022, 03:11:24 pm
content: Thanks for your detailed answerUnfortunately WebSockets aren't supported.I think we have to follow the other way: pure html.Is there any example for the php part that talks with the small buffering daemon (c code)?Thanks again for your time

author: ledtester
author_rank: Super Contributor
timestamp: December 09, 2022, 04:52:18 pm
content: Have a look athttps://socket.io/It can use HTTP long-polling for communication and there are PHP implementations (i.e. search for "php socket.io")You'll still have to write code to interact with the serial port but mayne this package will help:https://www.brainboxes.com/faq/how-do-i-control-a-serial-port-using-php

author: Nominal Animal
author_rank: Super Contributor
timestamp: December 09, 2022, 05:56:24 pm
content: The basic problem with doing it over pure HTTP is that any interactive Javascript will be run on the other side of the firewall, so a HTTP request must be made.Because of buffering in the HTTP server and proxies along the way, using a single request for multiple data chunks does not work (reliably at all).The model where the PHP script accesses the serial device directly only works for a query-response type interface, where the PHP writes something to the serial port, waits for a response (for a limited time), and reports the results.With the persistent agent, the PHP code can tell the agent "here's some more data to send, when you have the time", and ask "any new output to show?". So, the agent approach is incremental.The question as to how to connect between the PHP code and agent is an interesting one, since there certainly are many options to choose from. Named pipes have certain annoying properties (like blocking one end until another end has connected). Personally, I'd use a socket: either an Unix domain stream socket (/var/run/serial-agent-portname) or a TCP socket (127.0.0.1:8080), via PHP'ssocket_create(),socket_connect(),socket_shutdown(),socket_close().For simplicity, I'd have the PHP code send any output to serial port first, the shutdown the socket for writing, and then read any new incoming data from the serial port, and finally close the connection. The shutdown looks exactly like end of file to the other end. This way, you can have a separate thread in the C agentaccept()ing the new connection, reading all incoming data into a buffer, then decide whether to accept or reject it, and if accepted, send everything received from the serial port since the last update, and finally close the connection. (Remember, the PHP script already has the data from the web page in a GET or POST data chunk, regardless of whether the request was initiated via a form or via JavaScript.)It is very similar to the example code shown in e.gsocket_recv()page. The key is usingsocket_shutdown()to tell the agent "I'm done writing, now you tell me". When the agent has written all that it has received thus far, it closes the connection. This ensuressocket_read()will return an empty string (instead of blocking forever). (Similarly forread()/recv()on the agent side; they will return 0 when –– IF –– the PHP code shuts down the write side, and all data has been read.)If we look at the browser end, XMLHttpRequest seems the best way to do the update requests (that end up being served by the PHP script). While that does not have to match the data format between the PHP and the serial port agent, it sounds like a good idea to me: then, the PHP code is really just a relay, doing minimal to no processing to the (XML) data passed between the browser and the serial port agent. Well, it could also handle the user access control, either via cookie authentication or whatever other mechanism you're already using.In the browser, the XMLHttpRequests would be chained, so that when one completes, the next one is done say a second later.As bass-ackward it sounds, I'd really do some practical web page mockup at this point, with functional JavaScript able to "consume" and "produce" data via a dummy PHP script (that does not yet connect to anything). You'll be able to work out the kind of XML data packets and encoding for the data you'll want to use –– PHP can use raw binary I/O with the agent, but you'll want to encapsulate that into XML into something that is easy to show and consume in JS.Whenthatis more or less done, with all the features you need mocked up, it is actually quite straightforward to wire it up into a small C termios agent dealing with the serial port. As usual, it is the User Interface here that is the biggest amount of work (that us programmers prefer to avoid); and tackling it sooner rather than later will yield better results. (I'm sure you too feel the urge to just do the backend first, and then slap together a frontend that exposes its features.. but that is the Wrong way to do it, claims my personal experience. Mock up the UI first, find out the exact form of data transfers, and the rest will be much easier, and the complete result much more useful.)TL;DR: No, now is not the time to worry about the PHP-to-Agent back-end. You need to do the front-end first, so that you can discover exactly what kind of functionality and data interchange formats you need from that back-end. You may feel you already know, and you may want to push the UI to be the last thing you do because UIs are annoying, but that is an illusion, a danger luring many a programmer away from good, useful solutions.

author: voltsandjolts
author_rank: Unknown
timestamp: December 09, 2022, 06:00:51 pm
content: I think there are somewebsocketsolutions ready-to-use but websockets aren't an option for you.Other than that, I don't know of anything ready-to-use.Pythonflaskorbottleframeworks make it quite easy to create response web pages, since your restricted to GET/POST.Someone has done aserial interface for flaskalready (I've never tried it).

author: voltsandjolts
author_rank: Unknown
timestamp: December 09, 2022, 06:01:48 pm
content: Quote from: Nominal Animal on December 09, 2022, 05:56:24 pmThismight be an option.

author: Nominal Animal
author_rank: Super Contributor
timestamp: December 10, 2022, 10:38:00 am
content: Also, if you want to minimize server load, it is quite possible to replace the PHP code with a CGI or FastCGI one written in C.Quote from: voltsandjolts on December 09, 2022, 06:01:48 pmI understood that OP, DiTBho, already has a HTTP server running on port 80 on that machine.If not, then making the agent itself into a simple (threaded) HTTP server that listens to and responds to requests, opens up a lot of new possibilities. In such servers, the ability to handle being hammered with tarpitted requests is crucial (since lacking the ability makes Denial-Of-Service attacks trivial), so I personally would definitely write it in C or C++ myself. I do not trust the existing codebases, because of their poor track record regarding security and robustness against trivial scripted attacks.

author: DiTBho
author_rank: Super Contributor
timestamp: December 10, 2022, 11:58:14 am
content: Apache2 running on a mac-mini/ppc GNU/Linux

author: DiTBho
author_rank: Super Contributor
timestamp: December 10, 2022, 02:21:43 pm
content: Previous short answers, I wrote from the smartphone, terrible QWERTY kb.Quote from: Nominal Animal on December 10, 2022, 10:38:00 amNo problem for this, I'd like to write stuff in C, it's better than with PHP because I am more familiar with C.Anyway, I have zero experience and knowledge with CGI and this stuff.Any skeleton example around?

author: Nominal Animal
author_rank: Super Contributor
timestamp: December 11, 2022, 10:05:09 am
content: You'll want to usemod_fcgidwith Apache. The FastCGI protocol is documented for examplehere.The FastCGI module (mod_fcgid for Apache) will use an Unix domain stream socket to talk to the daemon. The daemon accept()s atransportconnection from the server, establishing the identity of the FastCGI module or server it is talking to, and exchanging some server-specific data, like the number of concurrent transport connections to that server. It is an interesting protocol, and for cases like this where there is no need toexecanother process, can significantly reduce the overall overhead.In particular, it would be possible to let more than one simultaneous client connect to the daemon, and even real-time chat with each other over the same facilities, making cow-orking on the same serial port device much more interesting. In other words, designed to cooperate on the single serial port.The UI could have e.g. write lock, so that only that specific client can send data to the serial port, but overridable via the chat interface.(If you have multiple serial ports, I recommend using a separate FastCGI for each, rather than making everything even more complex by having the daemon support more than one serial port at a time.)In Linux, you should findlibfcgi-binandlibfcgi-dev(andlibfcgi0ldbl) in the package manager, referring to the FastCGI library atgithub.com/FastCGI-Archives, which you can use from various languages, including C and C++. (It contains both a C and a C++ version, actually.)I'm rusty, but if you're not in a terrible hurry, I could see if I can put together something I might use myself.I must say that allowing a WebSockets connection over a dedicated port, say 8080, would be much better, both performance and programming-wise (on both the browser end and on the back end). Is the firewall completely out of your control, or could you consider adding a specific TCP port hole for WebSockets?

author: DiTBho
author_rank: Super Contributor
timestamp: December 11, 2022, 11:31:53 am
content: I am very busy with titanium bicycles; Birmingham 80s and Reynolds 2000s technology: very hard to find, lot of km just to pickup parts. I am also busy with a cycle computer I am implementing for a custom pseudo electro mecchanic rear derailleur (similar to Campagnolo Eps)Sweet dream, my-c used to burn the Eps firmware? Unsupported mpu, but it's in my wishlistYou know what has the highest priority, I'd like and need some real life example for the web app, but no problem.Unfortunately, the firewall is not under my control, kind of black box sent by a nasty company.It's an ADSL router, completely closed, and with a tamper-proof device. If you open the box, they will know it.That's why I say "nasty" campany. But hey? At least they offer a nice but cheap internet connection.I am still tempted to add a serial port in order to flash openWRT on it but it's not possible at the moment.One day ...

author: Nominal Animal
author_rank: Super Contributor
timestamp: December 11, 2022, 01:00:20 pm
content: Quote from: DiTBho on December 11, 2022, 11:31:53 amDon't you at least have access to its UI? To set 'customer' settings and whatnot, like explicit firewall holes?Assuming it is a stateful firewall (which it has to be if it can differentiate between HTTP and Websockets), it might allow connections through unprivileged ports like 8086, 48879 (0xBEEF), or 51966 (0xCAFE). Have you checked if those work? Or is it so tied down that only explicit protocols specified by the company are allowed to be used across the router? If you want, PM/email me and I can make an external connection from an IP address and port you'll know beforehand (I've got a publicly visible IP address myself), but definitely outside that telco's own network.This is important enough that personally, I would ask the company if they could allow the access. After all, it is a serial port on a single device, and WebSockets is a widely used service, and the port does not need to be privileged. I can guarantee it would be a better solution than FastCGI, plus when not used for the serial port, you could do other DIY experiments (like online realtime group chat and such) with it.

author: DiTBho
author_rank: Super Contributor
timestamp: December 13, 2022, 10:18:17 am
content: Quote from: Nominal Animal on December 11, 2022, 01:00:20 pmit has been authorised

author: Nominal Animal
author_rank: Super Contributor
timestamp: December 13, 2022, 11:24:51 am
content: I'll work out an example program and a web page. As I'm rusty, might take a couple of days, but please do poke me in case I haven't posted them here before the weekend.