altermatives to vt100? neater protocols? - Page 1
Forum section: electronics / general

author: DiTBho
author_rank: Super Contributor
timestamp: August 22, 2022, 11:56:52 pm
content: so, I put my money into the purchase of several vt-terms, a VT220 and a Digital DEC VT525.They are nice machines based on 8080 CPUs, they are nice when you just use them with Linux, and they are very useful when you program on the text-console, especially the VT525 with a faster serial.With VT100, ANSI escape codes for cursor control and other tasks became thede factostandard for hardware video terminals and later terminal emulators, and when you look at the protocol ... well, it's rather patchy, especially if you consider all the DEC addictions.I'd like to re-design the terminal in modern terms, supporting blocks like IBM 3.27K (but not too IBM-ish) to minimize the number of I/O interrupts required by transferring large blocks of data known as data streams with high an a 48Mbps optical fiber interface, udp/ip over Ethernet connection optionally available.So, a bit smarter than a dumb-terminal, not so much, but, absolutely of primary importance, the protocol must be designed in a super polished and neat way.I don't want to see complex ESC sequences with the quality of being so illogical, inconsistent, and unclear, that trying to implement it's fsm in HDL is practically more annoying than heaping and stacking stones in the mine,Suggestion, advice, all welcome

author: brucehoult
author_rank: Super Contributor
timestamp: August 23, 2022, 01:57:14 am
content: It seems to me that you are conflating two entirely different issues:1) the commands recognised / sequences sent by special keys, and2) the communications channel, whether 20 mA current loop, RS-232, SDLC, X.25 etc, synchronous or async, "line mode" or not.Waaay back 40 years ago when I was using such things, I seem to recall that the VT100 by default interpreted both ANSI ("VT100") commands and VT52 commands in the same mode. You could switch to ANSI-only using I think ESC < and to the mode that did both using ... uhhhh .. ESC [?2lI don't think VT100 had a pure VT52 emulation mode, that I can recall.When we wrote games on the PDP-11 or VAX, we used the VT52 codes to the maximum extent possible, because they are shorter and that made a huge difference at 9600 BPS. Especially the random cursor positioning using ESC Y row col (encoded in binary) was much shorter than the ANSI ESC [23;75H or whatever.Regarding communication channel: I remember at some point fairly early on a PDP-11/34 was re-purposed as a "LAT" to manage the async terminals and service their interrupts, and provide the VAX with nice packetised input and output. And to let you connect your terminal to any one of several VAXen too. That protocol is probably documented somewhere.Yeah.http://www.bitsavers.org/pdf/dec/ethernet/lat/AA-NL26A-TE_LAT_Specification_Jun89.pdfThat's dated 1989 but I was definitely accessing VAXen via this protocol by ... 1983 maybe.

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 23, 2022, 02:06:14 am
content: Why would you want to torture yourself implementing that in HDL (so I suppose using an FPGA, unless you actually plan on making silicon out of that), when even a small modern MCU would allow you to implement this with ease? Just wondering. (And yes, even with a VGA output without any additional ICs if you're using something as cheap as a RP2040!)As to the protocol, I've never looked at it closely enough to figure if it was really that bad. But sure if you don't need to be compatible with anything, you can devise a clean protocol, not based on clunky ESC sequences. And if you're targetting much faster interfaces than what was standard at the time, you can encapsulate everything in neat packets with nice headers and CRCs and whatnot.

author: westfw
author_rank: Super Contributor
timestamp: August 23, 2022, 07:15:47 am
content: x11?At one point “X terminals” (tektronix and NCD were two vendors) were a wonderful thing.They even supported a compressed Async protocol (XRemote) that made them useful over 9600bps dialup (and Cisco supported it in their terminal servers!)X runs over tcp/ip, of course.I think Stanford’s “V kernel” did similar things, but didn’t catch on anywhere. I suspect SUN siphoned away all of the developers.

author: DiTBho
author_rank: Super Contributor
timestamp: August 23, 2022, 08:59:41 am
content: Quote from: westfw on August 23, 2022, 07:15:47 amI own an original Tektronix xp-2xx-line with its netboot OS and licenseIt's MIPS-R3000 based @ 50Mhz with 16Mbyte of fastpage 72pin ram (must be <=50nsec), fully net-booting, VxWorks v5-based, built-in mwm Windows manager, telnet, a built-in superfast Nescape browser and a font client.First problem: it's pseudo color ---> forget everything GTK-based, only OpenMotif apps will work.Second problem: only HTML-v1 (1991) and HTML-v2 (1995) are supported ---> forget 90% modern web sitesHowever, it also adds two serial lines in case you need vt100 an vt200 emulation.And you don't need any "multi window" extension to operate with two sessions (you need >= vt420 for this), just open two of them, and connect each on a dedicated serial line, you have two physical serial lines, you can have up two sessions.Do you want more? Use telenet, connect to a trusted telent-to-ssh bridge (WR703? a router?) and you will have many many more.---I like it, I like the "thin terminal" concept, and I recently replicated with a modern PPC GNU/Linux + uclibc + nanoX + DWM + (Invisible Island version) xterm solution ...... just X11 and nanoX are toooooooooo big and complex to be implemented from scratch, and I won't for sure try to build my own.The funny here is: the PPC@200Mhz with GNU/Linux and X11 is 1.2x slower than X11 on VxWorks on a MIPS-R3000@50MhzKind ofLOL

author: DiTBho
author_rank: Super Contributor
timestamp: August 23, 2022, 09:14:24 am
content: Quote from: SiliconWizard on August 23, 2022, 02:06:14 amWell, Ania implemented a minimal vt100 in vHDL, and not only is it feasible but it's also cool and neat!Once defined the new protocol, the plan is to meet Ania to implement new things in HDL.Kind of interesting collaboration, you shouldn't underestimate this aspect.Social Engineering is always awesome.Plus, this project could even win an award for the next 2023 hack campingWe are talking about acool premium badge!!! and acase of Bel-cola!!!- -All the MPU-solutions I have ever seen around suck. Full of tricks and horrid things.When people have an MPU, the design tends to be illogical, inconsistent, and unclear, especially for those who suck at software design, and, worse still, have an horrid C-style programming.Yes, they write "sorry, I will polish and uncrustify in the next version"Which 99.999999% times meansgnegne ... forget it, it will never happenIt would be interesting to write the whole project in myC, but it only supports MIPS4++ and for sure I won't put a MIPS R18200 into a vt-term ...... I mean ... it would really be like sending aGrendizer robotto squash mosquitoes

author: DiTBho
author_rank: Super Contributor
timestamp: August 23, 2022, 09:28:35 am
content: Quote from: brucehoult on August 23, 2022, 01:57:14 amthere are two different needsOne ischaracter-oriented, good for the keyboard but you shouldn't limit the whole protocol to it otherwise you have to send every single char when you need to update the screen, and this way you have a lot of interrupts, and a very low throughputOne isblock-oriented, it offers higher throughput, and it's good when you have to send a whole sub-screen sessions, forms, or things like thatVT52, VT100, etc are char-oriented-only, and uses a char-device as communications channel, while I want to support a block-device like 48MBps fiber optic.The new protocol needs to minimize the amount of data transmitted, and minimize the frequency of interrupts by ensuring the CPU is not interrupted at every keystroke. I won't for sure send one character at a time like in VT100, worse still with a lot of ESC sequences to address row and column, and optionally also the char attributes.

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 23, 2022, 07:23:10 pm
content: Quote from: DiTBho on August 23, 2022, 09:14:24 amWell, everyone has a different concept of what is cool, but certainly it IS doable. My point was that it probably made little sense and would be harder to maintain.Quote from: DiTBho on August 23, 2022, 09:14:24 amMaybe, I dunno. Pretty much all commercial terminals have been implemented with MCUs/MPUs. That's certainly a workable approach.The "let's throw a FPGA at this project and see if it sticks" sounds like a weird engineering approach, but whatever floats your boat.Just because you have seen sucky implementations doesn't mean that you can't write a clean one (that stamtent sounded a bit like magical thinking), and certainly a VT100 terminal with say VGA output could be neatly implemented with just a RP2040 with even a lot of processing power to spare (yes you can implement a clean VGA output with the PIO). If you needed HDMI output, then sure you'd need additional parts though.Just a thought. This is ultimately down to your preferences and that's alright. But just don't necessarily convince yourself and others that this is the only right way of doing it.

author: DiTBho
author_rank: Super Contributor
timestamp: August 23, 2022, 08:15:30 pm
content: Quote from: SiliconWizard on August 23, 2022, 07:23:10 pmno, it's let define a finite state machine and see how we can make the protocol the neatest possible to minimize the finite state machine and get all the constrains satisfied.Which is better than throw a MPU at this project, write some shitty C code, and see if it sticks

author: brucehoult
author_rank: Super Contributor
timestamp: August 23, 2022, 10:49:07 pm
content: Quote from: DiTBho on August 23, 2022, 08:15:30 pmAll MPU+program combinations ARE finite state machines. And there's no reason to write shitty C code if you don't want to. Writing shitty asm code would be a lot easier than creating a huge FSM by hand. Or, you could write non-shitty C or asm code. That would be even better.It only takes about 200 LUT4s and 200 FF on an iCE40 and one RAM block to make an RV32I soft core that would be 100 times faster than the 8080 in a real terminal. On Xilinx it's about 130 LUTs I think.

author: DiTBho
author_rank: Super Contributor
timestamp: August 24, 2022, 12:40:13 am
content: Quote from: brucehoult on August 23, 2022, 10:49:07 pmA program on a MPU is NOT the minimal finite state machine possible.

author: brucehoult
author_rank: Super Contributor
timestamp: August 24, 2022, 01:47:46 am
content: Quote from: DiTBho on August 24, 2022, 12:40:13 amYou're not going to get the minimal FSM no matter what you do.Just the 24x80 screen with 8 bit chars and 4 attribute bits is 223040states already. before you even start counting interpreting incoming control sequences.

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 24, 2022, 03:49:14 am
content: Not sure I get DiTBho's point exactly - I think we can almost all agree that writing this in HDL will be harder than in software (so will take longer), and since software programming is very often discussed here, I think we can also all agree with the fact we can write decent and robust software. Especially for something like this which is not rocket science.But as I said, whatever floats everyone's boat.As I mentioned, I'd find this fun to implement on a RP2040, supporting multiple interfaces and using the PIO for a VGA (or similar) output. But if someone is more comfortable doing this in HDL, why not. I'm just pretty sure it will take more time.Now one could argue that implementing it in HDL might have more chances of being more robust - if just because you need to be extra careful when using an HDL, and you also are more likely to test your code using test benches and simulation than you are for pure software - in that sense, HDLs may be better to force you to do things right. But that's also debatable.

author: DiTBho
author_rank: Super Contributor
timestamp: August 24, 2022, 11:09:44 am
content: Quote from: brucehoult on August 24, 2022, 01:47:46 amNot the way you interpret it! You are not working at gate level but rather above the rtl level, you have vHDL and you can describe a finite state machines in the VDU stage that perfectly knows the concept of "row" and "column" and EA to correctly address the video ramIn Ania's vt100-lapop you have hardwaretwo-fish encryptiondirectly connected to the optical fiber circuits (Manchester encoded) and to the VDU.What arrives from the optical cable enters the protocol link engines, gets accepted or rejected (corrupted, go back n), gets payload extracted, enters the encrypt unit, gets decrypted, enters the VT100 engine (minimal set), gets interpreted, decomposed to operations for the VDU unit which address character to the video ram, and, on refresh event, you see it on the LCD.Very clean! No software on no Softcore implemented, all modules in pipelining, 100% vHDLAnia showed that all these units work in parallel and there are many bypassing circuits that would consume many more resources if implemented in software on general purpose CPUs, and commented that it's not because CPUs suck but rather because they are *generic* and filled with other things like pipelining, caching, etc. to go faster. Also, parallelism on a single core needs interrupts at least and context save/restore, which ... is good but not so much when compared to modules hardware pipeling.

author: DiTBho
author_rank: Super Contributor
timestamp: August 24, 2022, 12:05:09 pm
content: Meanwhile, talking about "multiprocessor programming"I just ordered this bookThe Art of Multiprocessor Programming, Revised Reprintby Maurice Herlihy, Nir Shavit

author: rstofer
author_rank: Super Contributor
timestamp: August 24, 2022, 05:29:49 pm
content: Quote from: DiTBho on August 23, 2022, 09:28:35 amActual keystroke interrupts occur at something less than 10 per second. 100 ms is a very long time. A 600 MHz Teensy 4.1 won't even notice the handler running. ARM Cortex M7https://www.pjrc.com/store/teensy41.htmlTelnet usually operates one character at a time (Character Mode) unless Line Mode is negotiated. Operator input obviously uses character mode since a single character response is often used (thinking of the response to 'y/N'). I don't see how you get around one character at a time when the message is only one character long. In terms of keyboard input, you have, at the driver level, no idea how long the message will be.There's a reason standards survive. They are useful and very nearly optimal in their application.

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 24, 2022, 07:02:34 pm
content: To each their own really! That's all we can say. But all this can clearly be written cleanly in pure software on a modest MCU. Only the encryption part may require a beefier one, but even this is probably not that much of a problem due to the typical data throughput of a text-based terminal. For a graphics terminal, things would be another story.

author: Jr460
author_rank: Regular Contributor
timestamp: August 24, 2022, 08:08:27 pm
content: I'm really not getting what you are trying to do and why, but I'll try to play along.In terms of less interrupts, while that is comm/serial controller's problem on the computer side, not the terminal's problem. Someone said LAT, well that was just the transport between the CPU and and the terminal server over Ethernet. You still came out of the terminal server at 19.2k bps or so to a terminal. LAT is going to wait for data until it has more to fill up an ethernet packet. I think the timer was something like 80 milliseconds, and then it had to send what not already had. The idea was to keep the delay below the time that the human eye / brain could detect the delay.How about this, go find an old VT1000. That is not a typo. I tw as monochrome X-term, so it had builtin ethernet, was based on on a 68K processor. And if you didn't want to do full blow X, you could pop open a bunch of vt100 compatible windows that natively did LAT over the network. Problem solved.As far as moving less data, look at the differences between VT100, 101, 102, 200 and 220, 240, 300 code. The later terminals allowed for 8 bit escape sequences which shorten many commands, also local line editing,, a character could be added or deleted mid line without re-painting the whole line. Also scrolling regions on the screen a set of lines or a block. If your program uses the right commands to the terminal, you can save a lot of output to refresh a screen.Or go real old school. have the terminal "screen" as part of system memoryand the video generator just cycle steals or does DMA from the screen buffer. I had an old S100 based board that did this. And also had a ROM on it that could be called passing it data, and it would emulate a vt100 and write to memory on the card for you.How fast do you really need data to come to fill a screen that is 80x24. What serial speed does it take to write the whole screen in under 80 ms? And faster than that and you will not see the difference. If you are blasting lines to screen just for it to scroll by and that is the delay in your program, then I'd say fix you program to only put out what need to see at the end.

author: DiTBho
author_rank: Super Contributor
timestamp: August 24, 2022, 09:09:57 pm
content: Quote from: rstofer on August 24, 2022, 05:29:49 pmyeah, not the best example, except when a keystoke causes a full page update, e.g. software vertical or horizontal scrolling.You have them with a text-editorlet's say, worst case, you have to update the whole screen, 80x25x2 (1 char, 1 meta), one page is 4K byte of ram(1), 4kbyte/s.gets no-dice with a slow uart connection at 9600bps (~ 900byte/sec)gets better with 115kbps (~10Kbyte/sec)gets excellent with Ethernet 10Mbps (~800Kbyte/sec)gets outstanding excellent with optic link 48Mbps (400Kbyte/sec)but here you may want larger screen, and also transport other stuff, so you want to compress and maximize the payload and get rid of all the extra things you have to transport for your terminalThe vt100-procol is ASCII based, not good for four pointsyou have to parse and decode fields, e.g. row=10 is written "1" "0" rather than of 0x0a, it's OK for MPUs, but an annoying waste of resources in vHDL because you have to write a mini fsm just to manage itit's less efficient than a binary protocol with fixed length fields and usually takes 2.5x the length, due to its ASCII nature + ESC encodingit's not orthogonal, there are ESC sequences that need their own dedicated sub-fsmit doesn't consider when you want to update clusters of chars in a single shotYou appreciate the last point especially with IBM 3.2K terminals, which are block-oriented, and whole page updating is one shot!(1) Ania's VDU text ram is 32Kbyte at the moment. It's a true dual port ram.

author: DiTBho
author_rank: Super Contributor
timestamp: August 24, 2022, 09:14:24 pm
content: Quote from: Jr460 on August 24, 2022, 08:08:27 pmyes, but starting from scratch I want to solve this too since who knows? I will probably use the terminal with super fast GNU/Linux MIPS64 board @ 1.4Ghz, perhaps with a 68hc11 @ 4Mhz ...let's make life easy on the server side too, if possible

author: DiTBho
author_rank: Super Contributor
timestamp: August 24, 2022, 09:17:07 pm
content: Quote from: Jr460 on August 24, 2022, 08:08:27 pmYup, I bought a LAT terminal, it's here in my lab. I'd like to *copy* the best form it

author: DiTBho
author_rank: Super Contributor
timestamp: August 24, 2022, 09:34:27 pm
content: Quote from: Jr460 on August 24, 2022, 08:08:27 pmYeah, and it's not supported by Ania's vt100-hdl, they are extensions, while Ania's one is a very basic vt100, like the first model appeared for the market.Nice feature addons, I can play with them with my VT525 and see how they are pretty and nice ... just, well they made the vt-protocol more spotted than a cow, and fist thing you have to do is probing your vt-term "which version are you?" and "which feature o you support?"flexible for different products that evolve year by year, but not good when you try to synthesize the minimal protocol for full features.

author: brucehoult
author_rank: Super Contributor
timestamp: August 24, 2022, 10:38:17 pm
content: Quote from: rstofer on August 24, 2022, 05:29:49 pmWell, yeah, but we used to have 50+ VT100s attached to a 1 MIPS VAX 11/780. 600 MHz Teensy is about 1000 MIPS. So the relative interrupt load on the VAX was something like a factor of 50,000 higher. That's why they started using old PDP-11s as terminal servers, to take that interrupt load off the VAX.

author: brucehoult
author_rank: Super Contributor
timestamp: August 24, 2022, 10:56:26 pm
content: Quote from: DiTBho on August 24, 2022, 09:34:27 pmYou haven't lived until you've programmed graphics using ReGIS / GiGi.Code:<ESC>P0pS(E)(C1)P[100,440]V(B),[+100,+0],[+0,-10],[-100,+0],(E)P[500,300],F(C[+100])<ESC>\That clears the screen then draws an outlined 100x10 rectangle, then a filled circle with radius 100.

author: Jr460
author_rank: Regular Contributor
timestamp: August 24, 2022, 10:59:49 pm
content: Quote from: brucehoult on August 24, 2022, 10:56:26 pmYep loved doing things with those graphics. Wrote tuns of stuff to take files in HP-GL or Tex 41xx format and output it as Regis.

author: Jr460
author_rank: Regular Contributor
timestamp: August 24, 2022, 11:08:21 pm
content: Quote from: brucehoult on August 24, 2022, 10:38:17 pmFirst you needed a better serial line controller, DMZ rather than a DMF, that will cut down on interrupts.I never saw the TT/TX driver or such imparting much load on a 785 or 8600. Also had people dial in at 19.2K and use Kermit to move data from mobile test stations why out in the great white north where they were luck to have a phone line. If you had the SPM+ package you could profile the time the CPU spent running code in system space. The TX/TT drivers never showed much of a load. Cluster comms and MSCP to HSCs, yea that could make a bump. Worst thing for interrupt time was page faults. Keep those under control and the machine ran great.

author: IanB
author_rank: Super Contributor
timestamp: August 24, 2022, 11:17:32 pm
content: Back in the 80's I programmed a whole set of complex colour text/graphics displays using a combination of ANSI and Tek 4014 graphics commands over a 19200 serial line running on a VAX minicomputer. I was able to get pretty good performance with nearly instant screen refresh. The key to it was to create an abstract graphics library to hide away the escape sequences and optimize the data stream sent to the terminal.VMS also had various WYSIWYG text editors running on VT terminals, such as EDT or TPU.One thing you would never do for good performance was to send one character at a time. You would always set up the terminal and then send a block of data.

author: DiTBho
author_rank: Super Contributor
timestamp: August 25, 2022, 12:34:49 am
content: Unfortunately impossible but it would be super cool... supporting svg in hardware.It can be limited to primitives, lines, arc, circles, ... Nothing more.Forget more complex curves.Anyway, enought to graph math 2D functions, I mean like a graphing calculator, which is something I'd like to have.Otherwise you can transport bitmaps. Ania has her personal logo encoded into the terminal, when she powers on and inserts the phassprede, the terminal shows a tiny image at the center of the screen.Nice bitmap ram, pre initialized Bram, it can be used to show things in the background.

author: DiTBho
author_rank: Super Contributor
timestamp: August 25, 2022, 12:39:24 am
content: Quote from: IanB on August 24, 2022, 11:17:32 pmCleever idea, now I want an ESC compositor library

author: DiTBho
author_rank: Super Contributor
timestamp: August 25, 2022, 12:52:53 am
content: Quote from: IanB on August 24, 2022, 11:17:32 pmYup!Years ago I wrote a text-editor. It runs on Linux and its interface uses ncurses.I think I will change its interface, so it will be the first application for the new-terminal protocol.It can be the ideal stressing-test application.Search, Go-To-Page, etc, are good examples of why you should send a block of data

author: DiTBho
author_rank: Super Contributor
timestamp: August 25, 2022, 01:10:09 am
content: The iconic DEC VT100 has an optional video card called "Selanar".It's as big as the motherboard, and full of chipsProbably it can help with graphics, but I don't exactly how much

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 25, 2022, 01:37:50 am
content: Well sure, with TTL logic of the 70s. The same kind of logic could be implemented with probably a few tens of LUTs on a very small FPGA.And as I said, even a $1 RP2040 with its PIO used cleverly can get you much, much better graphics than this thing ever could.

author: westfw
author_rank: Super Contributor
timestamp: August 25, 2022, 07:48:47 am
content: Tn3270? Telnet linemode option?I don’t understand what you plan to have this “improved” protocol talk to, since it’s been a long time since most operating systems have had anything other than a character-oriented “console” interface.

author: DiTBho
author_rank: Super Contributor
timestamp: August 25, 2022, 07:53:29 am
content: Quote from: SiliconWizard on August 25, 2022, 01:37:50 amNice on YouTube, then when you look at sources ... meh, C && cleverly use don't match with neat, especially with OpenSource.Technically I could even use the Timer Processing Unit of machines that have it aboard to implement softVGA@pixelclock 25MhzAlready done for spi, i2c, uart, PWM, etc, and it's notbit-banging, it's a set of dirty tricks on the cleverly use of a special piece of hardware called TPU that can be programmed to control PIOs, so nothing special under the bridge for me, I heard about it in 2001, and read lot lot of pages of manuals, experimented lot of compromises, spent lot of weeks to master the new toy.The $ 1 RP2040 probably means you'd better spend your weeks notuncrustifyingother people's things than getting too bored of wasting time on unclear manualsFor me HDL is the neatest solution here. No tricks. Just modules.You want to change the PHY? You don't need to hammer your head on the wall to get enlightened for new tricks to cleverly exploit the PIO in a different way, just change a module.I remember ... with Motorola' TPU, there was a manual of 500 pages, and still today there are some so unclear things that in going from PWM tosigma delta TPU channels... similar functionality, just a bit different timing, and I haven't yet understood what I did, and why? it works

author: DiTBho
author_rank: Super Contributor
timestamp: August 25, 2022, 08:29:51 am
content: Quote from: westfw on August 25, 2022, 07:48:47 amgood point, especially for LinuxIt doesn't need to support already existing stuff, like telnet.Likely, it will be this way:Protocol development and initial testingCode:. server side client side_________________ _________________| | | || Linux/R64 | | Linux/R64 || arm32/le | | arm32/le || | | || myTE | | Hterm || | | ncurses ||_________________| |_________________|\____________________________________________________/optical fiberOptical fiber here is achieved with a miniPCIe to PCI bridge + PCI optic adapter.R64 is a router based on MediaTek SoC. Two miniPCIe lanes.I already have to support it, I take the opportunity.Hterm here will work similarly to "minicom" or "screen", with its native support for ncurses so I won't need to write that stuff.Final applicationCode:. server side client side_________________ _________________| | | || Linux/R64 | | fpga || arm32/le | | || | | || myTE | | Hterm || | | VGA/ps2 ||_________________| |_________________|\____________________________________________________/optical fiberOptical fiber here is achieved with a dedicated module.Ania already wrote the vHDL driver, so I take the opportunity.myTE will be the first application exploiting Hterm, it's my text editor.Then I will write a remote-shell server for Hterm.Kind of stdin/out/err handler for Bash, but it will be only usable to issue line commands and see their line responses.forget things like htop and cmatrix for now.

author: DiTBho
author_rank: Super Contributor
timestamp: August 26, 2022, 01:50:39 pm
content: Today I am looking at tekonivel's DEC VT420 over a null-modem cable as Linux terminal.The VT420 terminal is used in VT220 emulation mode, because the “vt420” termcap entry doesn’t happen to be installed on my machine, anyway it's is used in that mode with 132 columns and 48 rows, page-length also set to 48 to get full-screen pages, and the cable is simple, and doesn’t provide hardware flow-control, and it works *just fine* for bash and text-line programs, but the speed is set quite low, to 9600bps, for text-editor, so in practice forget Nano, forget Joe, forget Vim+NerdTree, only Emacs is decently usable since it wisely updates only the parts of screen that have changed.That's interesting

author: IanB
author_rank: Super Contributor
timestamp: August 26, 2022, 06:37:51 pm
content: Quote from: DiTBho on August 26, 2022, 01:50:39 pmWhat about ^S/^Q flow control? You always needed that enabled in my experience for reliable operation.

author: DiTBho
author_rank: Super Contributor
timestamp: August 26, 2022, 07:12:04 pm
content: Quote from: IanB on August 26, 2022, 06:37:51 pmGood idea! soft flow control! Yup, the vtern supports it, thanks

author: DiTBho
author_rank: Super Contributor
timestamp: August 27, 2022, 12:59:15 am
content: Modifications for hterm: get rid of terminfo and termcapThe purpose of a terminfo and termcap files (one file for each terminal/mode) is to let programs (e.g. Nano, Vi, Vim, Emacs, Joe, htop, bmon, etc ...) running in the terminal know the capabilities of the terminal.Code:toe -awill show many files, one for each supported terminal/modeCode:toe - table of (terminfo) entriesSYNOPSIStoe [-v] [-ahuUV] file...DESCRIPTIONWith no options, toe lists all available terminal types by primary name with descriptions. Filearguments specify the directories to be scanned; if no such arguments are given, your default ter-minfo directory is scanned. If you also specify the -h option, a directory header will be issued aseach directory isThere are other options intended for use by terminfo file maintainers:-a report on all of the terminal databases which ncurses would search, rather than only the firstone that it finds.Termcap and terminfo, basically answer questions like:- Does it support colors? How many?- How many column? How many rows?- How special keys are mapped?etc...A VT100 terminfo should make programs assume an 80 column monotone screen and a keyboard with 10 function keysVT100 compliance is why the F11 .. F12 keys are traditionally not used by console programs, which is really annoying when programs using "ten" function-keys (like mc) on a terminal lacking ten usable function-keys and YOU have to accept an alternate such as escape-0, escape-1, etc because there's no intuitive "escape-11".- no way, this has to go! -The protocol of Hterm is fully auto-probing. When a program needs to know something it has to ask directly to the terminal instead of looking for some file in /etc/term... or /usr/share/term...And you don't have to compile any terminfo file with program like "Tic"

author: DiTBho
author_rank: Super Contributor
timestamp: August 30, 2022, 07:25:46 am
content: CompressionSo, yesterday I was considering string compression for large cluster of chars, which can be useful during page_update or page_scroll to minimize the effective number of data to be transmitted.Hterm supports this kind of thingsAnia has designed a very simple, very small and self-contained string compression algorithm, Zeda(1) has provided us with a dictionary which contains what she thinks is the best combination of two letters used in English, and we analyzed how the compression ratio behaves with real text pages.Ratio = length(compressed string) / length(original string) * 100Average Ratio: 59 of 100Best Ratio: 55 of 100 <------------ theoretically it is 50 of 100Worst Ratio 79 of 100 <----------- theoretically it can be 100 of 100, ZERO compressionnot as good as the compression you will get out of a zip file or something similar to that, the maximum compression that can possibly be achieved is 50%, and her classmate Zeda said that, depending on the *nature* of what you are compressing, you may find that a different dictionary that may give better performance.The dictionary Zeda gave us seems to work reasonably well with pages of "C source" text and pages of "English articles".just ... is string-compression worth it on a 48 Mbps link? big question

author: brucehoult
author_rank: Super Contributor
timestamp: August 30, 2022, 08:10:07 am
content: Quote from: DiTBho on August 30, 2022, 07:25:46 am48 Mbps? 6 MB/s?lz4 with default settings gets 2.1:1 compression on the "Silesa Corpus" at a speed of 625 MB/s on one core of an i7-3930K @ 4.5 GHz.That's on a 2011 CPU that does memcpy() at 7300 MB/s.If you only need 1.8:1 compression it will do that at 911 MB/s.If you want 2.7:1 compression (-9) that slows lz4 down to compressing only 34 MB/s. At that point you're better off using zstd -3 which gets 3.16:1 compression at 200 MB/s.lz4 decompression is at over 3200 MB/s on the same CPU, regardless.lz4 is a small and simple algorithm. It's hard to think of a reason to NOT give it a quick run over data rather than sending raw data.I don't have figures to hand for 2022 CPUs, but they will be better.

author: brucehoult
author_rank: Super Contributor
timestamp: August 30, 2022, 08:35:27 am
content: So, I just tried on my 2020 M1 Mac Mini, using lz4 from homebrew211938580 Silesa corpus total size100934427 lz4 default, saved 52.37% in 0.402s CPU = 502 MB/s.oh.Oops. That was an x86_64 lz4. So that was being emulated.Let me build a native one.ok, now it's 0.378s CPU time, identical output size. So that's 534 MB/s.Your comms link was 6 MB/s, right?

author: DiTBho
author_rank: Super Contributor
timestamp: August 30, 2022, 11:31:57 am
content: Quote from: brucehoult on August 30, 2022, 08:10:07 amyup, and it's 5Mbyte/s payload due to transport overhead and synchronization.Quote from: brucehoult on August 30, 2022, 08:10:07 amyup, but it needs to be implemented in hardware, and it should be simple and small.lz4 is very nice, more complex than our simple string compressor, but more performative

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 30, 2022, 07:20:35 pm
content: I have reimplemented this one (in a cleaner way) based on LZ77:https://github.com/banebyte5115/xlzIt's very fast (faster than LZ4) and gives good compression ratios.Implementing it in HDL shouldn't be too hard, it's a relatively simple algorithm. I might do it one of these days.

author: brucehoult
author_rank: Super Contributor
timestamp: August 30, 2022, 09:36:31 pm
content: Quote from: DiTBho on August 30, 2022, 11:31:57 amIf it's implemented in hardware then it's obviously going to be much faster. If you can do VT100 in pure state machine hardware without a CPU then lz4 is a doddle.After all, lz77 and lz78 were originally implement in hardware, in IBM disk drives.

author: DiTBho
author_rank: Super Contributor
timestamp: August 31, 2022, 12:54:42 am
content: Quote from: brucehoult on August 30, 2022, 09:36:31 pmyup, so Hterm will have the choice to negotiate the compression.query, do you have Zeda's compression? yes/no (here we need to find a nice name for her algorithm)query, do you have lz4 compression? yes/nonone of them means -> no compression

author: DiTBho
author_rank: Super Contributor
timestamp: August 31, 2022, 10:40:22 am
content: Code:msg Original=[suppa suppa, suppaman can fly faster than a fly and can jump higher than a earthworm blablabla better stronger than ratman]msg Restored=[suppa suppa, suppaman can fly faster than a fly and can jump higher than a earthworm blablabla better stronger than ratman]Original msg Length=122Compressed msg Length=74Restored msg Length=122Compression Ratio=60 of 100(Zeda's algorithm v1, dictionary ULK-11)Code:msg Original=[suppa suppa, suppaman can fly faster than a fly and can jump higher than a earthworm blablabla better stronger than ratman]msg Restored=[suppa suppa, suppaman can fly faster than a fly and can jump higher than a earthworm blablabla better stronger than ratman]Original msg Length=123Compressed msg Length=100Restored msg Length=123Compression Ratio=81 of 100(xlz algorithm)Ummm, the above is just a short example-string, I have to run more real full screen-page and check average compression rate, min, max, but it seems Zeda's wins for simplicity.No doubt about.Then we also have binary-pages, containing meta attributes for each char.Color foregroundColor backgroundBlinking?Underlined?etc

author: brucehoult
author_rank: Super Contributor
timestamp: August 31, 2022, 11:06:27 am
content: Your "Zeda's algorithm" is using a pre-loaded dictionary which must be previously shared/transmitted between the two ends.Other algorithms are starting from zero shared knowledge.Other algorithms (e.g. lz4 "-D file" flag) can also be initialised with a dictionary known to both sender and receiver, and will obviously do much better on short texts if they are.For example if I save this entire discussion thread as a .html file (193793 bytes, at present), and then use it as a dictionary for lz4, then your 122 byte message is compressed to 63 bytes with default settings, or 29 bytes with "-9".

author: DiTBho
author_rank: Super Contributor
timestamp: August 31, 2022, 10:20:12 pm
content: Quote from: brucehoult on August 31, 2022, 11:06:27 amZeda is Ania's classmate. She decided to join our hacker team, so now I have a team of two girls, like James Bond with a blonde and a brunetteYes, her algorithm needs a pre-loaded dictionary 320 byte sized.It should be loaded into BRAM.It's designed to map this charsetCode:charset{' ','!','"','#','$','%','&',''','(',')','*','+',',','-','.','/','0','1','2','3','4','5','6','7','8','9',':',';','<','=','>','?','@','A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z','[','\',']','^','_','`','a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z','{','|','}','~','\n'};So, she reduced a subset of ASCII-8bit into 96 possibilities, and '\0' is not included.I grabbed this topic' page to make a test with a bigger stream (16Kbyte):Code:Compressed msg Length=11074Original msg Length=16804Restored msg Length=16804Compression Ratio=65 of 100(Zeda's algorithm)Code:Compressed msg Length=9470Original msg Length=16804Restored msg Length=16804Compression Ratio=56 of 100(xlz)Not bad, considering its simplicityI will also implement xlz

author: DiTBho
author_rank: Super Contributor
timestamp: September 01, 2022, 12:22:57 am
content: with the same testing fileCode:./lz77 -c -i guinea-pig.txt -o guinea-pig.lz -s 2048lsprettysize guinea-pig.*10. 4 Kbyte guinea-pig.lz16. 8 Kbyte guinea-pig.txt(lz77, Git Repohere)

author: SiliconWizard
author_rank: Super Contributor
timestamp: September 01, 2022, 02:03:17 am
content: Quote from: DiTBho on August 31, 2022, 10:20:12 pmYes! I was also impressed when I tried it. Pretty cool.

author: westfw
author_rank: Super Contributor
timestamp: September 04, 2022, 01:28:18 am
content: I don't think I understand the point of speeding up an interface/protol that already goes faster than the data can be displayed on most displays, and MUCH faster than it can be consumed by a user...I guess it's fun. But...

author: DiTBho
author_rank: Super Contributor
timestamp: September 04, 2022, 02:34:33 am
content: Quote from: westfw on September 04, 2022, 01:28:18 amIt also needs to transport other stuff.Including bitmaps.

author: DiTBho
author_rank: Super Contributor
timestamp: September 05, 2022, 04:08:09 pm
content: Quote from: westfw on September 04, 2022, 01:28:18 amreason2, today a new member joined the team and proposed to make a vintage CPU version of hterm.His hardware has 10Mbit/s ethernet, hence it will use a subset of the hterm protocol and see built-in compression helps.

author: SiliconWizard
author_rank: Super Contributor
timestamp: September 05, 2022, 08:32:26 pm
content: Odd vintage world.