Command languaje for tiny microcontrollers - Page 1
Forum section: electronics / general

author: Picuino
author_rank: Super Contributor
timestamp: October 09, 2022, 03:51:32 pm
content: An interpreted language allows us to program and test with the microcontroller faster than with compiled languages, but there are not many options to choose from. Here I present a few and the solution proposed by zserge.LUAIs a good interpreted language. Powerful and flexible.It occupies little space (200kB flash) but it is too much for small microcontrollers.On larger microcontrollers, its garbage collection system makes it not recommended for real time applications.FORTHIs the opposite. Powerful, fast, occupies only 2kB of flash and is suitable for real time systems.The problem it has is that it tends to create read-only programs. Very complicated for an average user.TCLAn extendable language, born out of frustration with having to build custom command languages again and again, is in fact very simple.PARTCLTCL Adapted to microcontrollers:https://zserge.com/posts/tcl-interpreter/https://github.com/zserge/partclFeatures:~600 lines of "pedantic" C99 codeNo external dependenciesGood test coverageCan be extended with custom Tcl commandsRuns well on bare metal embedded MCUs (~10k of flash is required)Code distributed under MIT license

author: Picuino
author_rank: Super Contributor
timestamp: October 09, 2022, 04:08:06 pm
content: Another TCL interpreter in ~500 lines of code is Picol:Code:http://antirez.com/picol/picol.c.txtWiki:https://wiki.tcl-lang.org/page/Picol

author: tellurium
author_rank: Frequent Contributor
timestamp: October 09, 2022, 04:13:12 pm
content: Another option:A very cut-down JavaScript implementation:https://github.com/cesanta/elkless than 1500 lines of code,takes only ~100 bytes for the VM,can run on Arduino Nano with 2k RAMzserge, in fact, was one of the contributors of elk.

author: RoGeorge
author_rank: Super Contributor
timestamp: October 09, 2022, 04:21:02 pm
content: How small is tiny?Is MicroPython small enough?Quotehttps://github.com/micropython/micropython

author: Picuino
author_rank: Super Contributor
timestamp: October 09, 2022, 04:25:01 pm
content: MicroPython is too large for 8-bit microcontrollers. And the garbage collection system does not allow using it for real-time systems.Tiny in this context can mean 10-20kB at most, to make room for more program parts and customizations.

author: nctnico
author_rank: Super Contributor
timestamp: October 09, 2022, 04:39:19 pm
content: Quote from: Picuino on October 09, 2022, 04:25:01 pmWhy would you want to use a scripted language for realtime systems? And you really need to define what is realtime. Once per second is also realtime but not a problem to achieve using a scripted language.IMHO a much better approach is to let the scripted language do housekeeping + high level logic and run regular C code to do tasks that need to be done quickly in parallel. I have been working on a modified version of LUA that can be used together with C software that is organised as a super-loop. Still, LUA takes around 80kB of flash on ARM and a minimal script already needs 12kB of RAM (including the LUA compiler). In principle it works but I want to add some debugging facilities to it so you can do on-target debugging using a command line interface.

author: Picuino
author_rank: Super Contributor
timestamp: October 09, 2022, 04:56:22 pm
content: I want a scripting language to make programs that test commands made with C code.I program low level routines in C and test them until they work.Then I want to make a program that calls this low level routines as needed, in which I will make many changes to tune it and that I do not care that it is slow in execution, but flexible and fast when programming it.In the end I can convert that code to C once it is well debugged.Another solution I have thought of is to use an interpreted language in the computer, which sends codes through the serial port. Each code corresponding to a call to a routine or an argument. And that sometimes the microcontroller responds by sending return messages to the computer.In short, I want to have a scripting language to handle small microcontrollers.Realtime may be 0.1s or 0.01 seconds, depending on the context.

author: globoy
author_rank: Frequent Contributor
timestamp: October 09, 2022, 05:03:46 pm
content: I'm curious, can you give an example of what you do with one of the small TCL implementations? What kind of system it's running on? I took a quick (but not exhaustive) look at their source and it doesn't appear they do things like interface with a micro's peripherals (perhaps without the addition of custom commands). They don't seem to have native filesystem access (for small systems with a filesystem in flash or on an SD Card). I've used TCL in the distant past as a scripting language on unix/linux systems. I am probably being dense, but I'm missing how they help on low-end systems like 8-bit micro-controllers.

author: Picuino
author_rank: Super Contributor
timestamp: October 09, 2022, 05:04:16 pm
content: Another one for Arduino.TinyBasic Plushttps://github.com/BleuLlama/TinyBasicPlus

author: Picuino
author_rank: Super Contributor
timestamp: October 09, 2022, 05:07:04 pm
content: Quote from: globoy on October 09, 2022, 05:03:46 pmI am not sure how to adapt TCL to microcontrollers. All arguments are strings and that doesn't get along with arguments that must be integers or bytes or bits. You have to do a conversion which is not trivial.Maybe the old BASIC that was in the early personal microcomputers is a better option.Quote from: globoy on October 09, 2022, 05:03:46 pmI'm thinking in Arduinos (Atmega328P) or equivalent PIC microcontrollers. 8bit microcontrollers with ~32kB flash and 2kB RAM.

author: Picuino
author_rank: Super Contributor
timestamp: October 09, 2022, 05:30:24 pm
content: Quote from: tellurium on October 09, 2022, 04:13:12 pmI have been testing ELK v2.2.0 for Arduino UNO and a minimal program (blink led) occupies almost all the Flash memory (30010 Bytes).It's a bit big for small micros.

author: tggzzz
author_rank: Super Contributor
timestamp: October 09, 2022, 05:30:36 pm
content: Quote from: Picuino on October 09, 2022, 04:56:22 pmDon't over-think the problem; just try something and see.Personally I'd go with Forth, since it is readily available, without "brick walls", well documented, and simple.You will be peeking/poking a device to verify the hardware and your understanding, then combining peek/pokes to make useful composite operations, then combining those into test sequences. That's bottom-up programming, and there's nothing wrong with that!Forth is ideally suited to bottom-up programming.

author: Picuino
author_rank: Super Contributor
timestamp: October 09, 2022, 05:32:45 pm
content: We will have to give it a chance.NanoFORTHhttps://www.arduino.cc/reference/en/libraries/nanoforth/

author: fchk
author_rank: Frequent Contributor
timestamp: October 09, 2022, 05:33:24 pm
content: Do you know FICL?https://ficl.sourceforge.net/very fast, very mature.fchk

author: Picuino
author_rank: Super Contributor
timestamp: October 09, 2022, 05:44:25 pm
content: NanoFORTH v1.4.0Multitasking.blink programFlash = 14184 bytes (43%)RAM = 278 bytes (13%)I am not able to do a simple for loop of 10000 steps to evaluate its speed.That's the problem with FORTH, it reminds me the HP48 calculator and its RPN system. Somewhat complicated.

author: Picuino
author_rank: Super Contributor
timestamp: October 09, 2022, 05:46:47 pm
content: Quote from: fchk on October 09, 2022, 05:33:24 pmNo. Thank you.It appears to be an implementation of FORTH for 32-bit and 64-bit microprocessors, but fairly easy to port to other systems.

author: SiliconWizard
author_rank: Super Contributor
timestamp: October 09, 2022, 05:50:19 pm
content: Regarding Lua:- You may want to have a look at eLua if you haven't already:http://www.eluaproject.net/(not sure it is still maintained though, unfortunately.)- Another option if the Lua language is attractive for your application, but you're OK with compiled rather than interpreted (which actually can be a huge benefit unless you need an interactive interpreter) is Nelua:https://nelua.io/. It's a fantastic project. Now of course, it will require some work to make an environment work for your particular target - it's not a "plug-and-play" solution.

author: rstofer
author_rank: Super Contributor
timestamp: October 09, 2022, 06:04:32 pm
content: Back when dinosaurs roamed the earth, I wanted a higher level language for testing new 8085 projects. The Intel monitor was a start but was console interactive, not scripted.Along came Palo Alto Tiny Basic (thanks to Li-Chen Wang). I could write code for any hardware I wanted to include in the system. It was far from perfect but it used less than 2k of PROM. Google shows quite a bit has happened since the early '80s and Tiny Basic is still around. One addition was an interface to CP/M which added a file system and standardized console IO.When I would create a new system, the first software would be the Intel Monitor just to see if memory was working properly and perhaps to test serial IO. Very soon thereafter, I would implement Tiny Basic. Sometimes I would load it with the Monitor, other times I would burn it into the PROMs. Source is still available for several versions.https://en.wikipedia.org/wiki/Tiny_BASIC

author: Picuino
author_rank: Super Contributor
timestamp: October 09, 2022, 06:07:26 pm
content: Thanks. Perhaps a very good option for 32bit micros.I'm trying to use nanoFORTH on Arduino.Code:: BLK 1 5 OUT 0 5 OUT ;BLKProduces a pulse of 50us duration.Code:: BLKS 0 FOR BLK NXT ;100 BLKSReset the microcontroller. It does not work. I don't know why.

author: Picuino
author_rank: Super Contributor
timestamp: October 09, 2022, 06:13:12 pm
content: Code:: BLK 1 5 OUT 0 5 OUT ;: FBK 0 FOR BLK NXT ;100 FBKNow it really works. It has a cycle time of 150us (6660Hz). Not too fast. Similar to TinyBASIC

author: rstofer
author_rank: Super Contributor
timestamp: October 09, 2022, 06:13:33 pm
content: Quote from: tggzzz on October 09, 2022, 05:30:36 pmBack in the late '70s, I went to a factory where they built disk drives and SCSI controllers. They used FORTH as their lab language of choice. The code was written bottom up so they could add a new test by just invoking some previously defined words for primitive operations. They were highly enthusiastic for this approach to testing.

author: pqass
author_rank: Super Contributor
timestamp: October 09, 2022, 06:14:48 pm
content: I played withhttps://www.flashforth.com/a couple of years ago. It seems nice.Assembler source (for avr, pic18, pic24), ready to burn hex file (for 328,32u4,2560), and docs/tutorials are provided.Can do multi-tasking, spi, i2c, bare metal ops, ...https://pajacobs-ghub.github.io/flashforth/ff5-quick-ref.htmlhttps://pajacobs-ghub.github.io/flashforth/elements-of-flashforth-5.htmlhttps://pajacobs-ghub.github.io/flashforth/ff5-tutorial-guide.html

author: Picuino
author_rank: Super Contributor
timestamp: October 09, 2022, 06:24:00 pm
content: TinyBASICFlash: 8020 bytes (24%)RAM: 1375 bytes (67%)Code:>10 FOR I = 1 TO 10000>20 DWRITE 5, HIGH>30 DWRITE 5, LOW>40 NEXT I>RUNCycle time of 550us (1820 Hz)

author: Picuino
author_rank: Super Contributor
timestamp: October 09, 2022, 06:30:57 pm
content: Quote from: pqass on October 09, 2022, 06:14:48 pmGood option, but it overrides the Arduino or PIC bootloader where it is installed.

author: tellurium
author_rank: Frequent Contributor
timestamp: October 09, 2022, 07:05:15 pm
content: If your goal is to make a well tested code, why not developing in С but on a linux/mac/windows, and abstract/stub the hardware calls.Basically, write a unit test in C, stubbing the HW, and when it is well tested, your C code is ready to be moved to the uC.

author: nctnico
author_rank: Super Contributor
timestamp: October 09, 2022, 07:30:00 pm
content: Quote from: tellurium on October 09, 2022, 07:05:15 pmThat is the most sensible thing to do indeed. Avoid debugging software on a microcontroller as much as possible. I wrote an ISDN protocol stack that way using Borland C under MS-DOS.

author: tggzzz
author_rank: Super Contributor
timestamp: October 09, 2022, 07:39:48 pm
content: Quote from: nctnico on October 09, 2022, 07:30:00 pmYup. I triumphantly reinvented that 40 years ago, for a Z80 with an ICE, and PDP-11 running Microsoft's decent operating system: Xenix. The boundary was events generated by the hardware in one direction, actions in the other, and an FSM using cooperative scheduling running under Xenix or bare Z80 as appropriate.But where that isn't possible, hardware testing using Forth is a decent alternative. The observant will realise thatdepressingly little has changed. Z80 <-> atmega328, ICE <-> JTAG, Xenix <-> Linux. Sigh.

author: SiliconWizard
author_rank: Super Contributor
timestamp: October 09, 2022, 07:47:17 pm
content: Quote from: nctnico on October 09, 2022, 07:30:00 pmI always recommend that, especially for non-trivial parts of the code, like particular processing, algorithms, etc.

author: westfw
author_rank: Super Contributor
timestamp: October 09, 2022, 09:41:37 pm
content: QuoteI'm not sure that's true any more. The whole cross-compile/simulate/burn process has gotten pretty quick and convenient, and the PC-side tools are much more capable than any embedded interpreter editing capabilities, which is a pretty big advantage once your programs get past "trivial" in complexity.QuoteI think for interpreted languages capable of running on a 32k/2k microcontroller, you're pretty much stuck with Forth-like languages, or BASIC-like languages. There are a lot of choices for either one, depending on ... I dunno what. I've used AMForth (which hasn't been mentioned yet - That makes at least four?) on Arduino hardware.Interpreters are able to optimize particular operations that you may be interested in. Remember "The BASIC Stamp"? It ran god-awful slow (by todays standards. About 1000 lines/s), but could do 9600bps async Serial output on any pin, and had a bunch of other low-level commands (not "standard BASIC") that made it useful for all sorts of things.I've seen tiny LISP, too. Hah hah!

author: florian-x
author_rank: Contributor
timestamp: October 09, 2022, 09:51:48 pm
content: Tasmota project uses Berry language in their esp32 build:https://github.com/berry-lang/berry. I plan to have a close look at it (some time...).

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 09, 2022, 11:53:26 pm
content: Quote from: tellurium on October 09, 2022, 07:05:15 pmExactly.I myself put each one of them in their own directory, with a small README file describing it, and file it away for later reuse. If it contains math, I drop LibreOffice .odt and .pdf there too (although I really should use LaTeX and have a small scriptlet for quick viewing.. it's been too long since I used proper LaTeX; MathJax comes closest nowadays).I do that for all key parts, including unit testing new algorithm implementations and verifying an approach works, for both systems stuff and embedded/microcontroller stuff. Not only does it reduce my cognitive load (because once tested and marked so, I don't need to worry about the details), but coming back to the same thing later on, you can quickly get back on track (given the README and math), and can discover new stuff. The last time I did that was just a couple days ago, wrt.indexing binary search treesI posted in the Fun for nerds thread.For the really complicated stuff, like user interfaces for microcontroller stuff, I actually write a test harness (in Linux) that implements roughly the interfaces provided by the microcontroller hardware. One useful trick is to run a Gtk+ user interface in the main thread, with its actions modifying variables atomically (usingGCC __atomic builtins), and interrupts simulated using thread-targeted POSIX realtime signals (SIGRTMIN+0toSIGRTMAX) –– they're not; they're just called that, and are just queued unlike ordinary Unix signals –– sent viapthread_sigqueue(). You can use a single signal handler and dispatcher that verifies the signal came from the process itself (siginfo->pidis enforced by the kernel), and use the payload value to indicate the "interrupt". For the UI, I like to use the GtkBuilder .UI interface, so you can use an external XML .ui file to describe the user interface and buttons, editable in e.g. Glade in WYSIWYG manner. Makes for a rather effective rapid development environment.(Signal delivery will interrupt whatever else that thread is running, just like interrupts do on real hardware. Technically, POSIX says that of the standard C library, onlyasync-signal safefunctions can be used in signal handlers, but for a microcontroller simulated environment this is not a big deal, because there usually isn't much of standard C library available anyway.)

author: newbrain
author_rank: Super Contributor
timestamp: October 10, 2022, 08:01:58 am
content: Quote from: Picuino on October 09, 2022, 06:13:12 pmAh, I was wondering why you first attempt did not work: nanoForth is not a complete FORTH, it has some limitations, 3 letters words being one.Which does not learn from the past: a common practice (not ANSI compliant) was saving only the first 3-5 characters of a word, and its 'real' length - of course there will be conflicts, but they are usually quite less than expected - quit an improvement from the user side, with minimal memory penalty.As for the speed, they are paying the penalty of token threaded code.Indirect or direct threading implementations are usually much more efficient, at the cost of being non-portable.The code size is also quite large - probably due to C++ use plus token threading, which needs a more complex VM: a direct or indirect threading FORTH would just need an inner interpreter (the execution engine, a handful of assembler instructions) and an outer interpreter (the user facing part, parsing and dictionary, most of it can be written in FORTH itself).

author: brucehoult
author_rank: Super Contributor
timestamp: October 10, 2022, 10:45:48 am
content: Quote from: newbrain on October 10, 2022, 08:01:58 amBased on that description the original code should work, as BLK and BLKS have the same initial three characters but differ in length.Perhaps it is only using the first three characters and not the length. That would make sense if the 4th byte was the assigned token, and you can only have 256 words defined in the entire program, including built in ones.I wonder how they distinguish between machine code words, token words in ROM, and token words in RAM. Perhaps just numeric token ranges, since the former two are fixed at system compile time. AVR can't run machine code from RAM, so more can't be added at runtime, words added by the user at runtime can only be token threaded and in RAM.On an ATMega328 16 bits is enough for either a RAM (byte) address or ROM (word) address. True even on a Mega 1280, but not 2560 (RAM ok, but ROM is too big).

author: newbrain
author_rank: Super Contributor
timestamp: October 10, 2022, 04:26:40 pm
content: Quote from: brucehoult on October 10, 2022, 10:45:48 amYep,that's what I meant: nanoFORTH is not learning from the past history of resource constrained FORTHs.I did not dig too much into it to say how it handles its threading.

author: Picuino
author_rank: Super Contributor
timestamp: October 10, 2022, 05:18:06 pm
content: Another solution could be to make a command interpreter on the PC.Or simply a language (Lua, Python, interpreted C) that sends instruction codes over the serial port. At 115200 baud it could send 5760 instructions per second with 1 byte argument, faster than most interpreters.On the microcontroller side, you can simply program a loop that reads the command received by the UART and executes it by calling the corresponding function.It would be fast, flexible, occupies little memory and you can make the program as large as you want on the PC, with no memory limitation.

author: paf
author_rank: Regular Contributor
timestamp: October 10, 2022, 06:05:32 pm
content: Quote from: Picuino on October 10, 2022, 05:18:06 pmThat already exists (at least in Forth) and is called a Tethered Forth.More Info:https://dl.acm.org/doi/pdf/10.1145/122089.122091https://pages.cs.wisc.edu/~bolo/shipyard/3ins4th.html

author: Picuino
author_rank: Super Contributor
timestamp: October 10, 2022, 06:35:04 pm
content: Great development!!

author: Kleinstein
author_rank: Super Contributor
timestamp: October 10, 2022, 08:54:37 pm
content: Writing a program for an interpreter may be marginally faster than for a compiler. However with modern PCs and the small µC programms the cross compiler is quite fast unless there is something very wrong set up.The big drawback of interpretet code is that it usually misses many of the error checks / warnings that a compiler can do. A Python program may fail with a syntax error when handling a rare case.One could consider a different langish than C (e.g. maybe ADA or a Pascal variant), but no real advantage in not using a complier as an extra checking instance and giving a little more speed.

author: SiliconWizard
author_rank: Super Contributor
timestamp: October 10, 2022, 09:03:25 pm
content: Yeah. The real question here is to ask oneself: why do I need an interpreted language on a MCU? And from there would come possible solutions. If you don't have an answer to this question ("just because" doesn't count), this is kind of pointless to answer.

author: ogden
author_rank: Super Contributor
timestamp: October 10, 2022, 09:53:52 pm
content: Quote from: SiliconWizard on October 10, 2022, 09:03:25 pmRight. For me interpreter language for tiny, thus low cost MCU does not make any sense - because low cost means optimization of code size out of your mind. Using interpreter would be cardinal sin in such case. Thou.. during prototyping when one is not sure about logic/physics/etc of final product or product is deemed to be customized over time like science equipment - he could consider to take powerful enough MCU, put interpreter on it - so engineers/scientists or just customers can rapidly tune/change design without help from pro coders.

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 10, 2022, 10:54:29 pm
content: For case insensitive languages with ASCII alphanumeric identifiers (that begin with a letter), there are 34,658 identifiers up to three characters long, 1,247,714 up to 4 characters long, and 44,917,730 up to 5 characters long. The initial character takes 4.70 bits, and each succeeding character 5.17 bits. You can fit any one- or two-character identifier in ten bits, or any three-character one in just over 15 bits. Up to four characters fit in 21 bits, five in 26 bits, and six in under 31 bits.If identifiers are case-sensitive, start with a letter, and contain letters, digits, underscore, and one other character (say '.'), the identifiers are basically base-64 digits (except the initial digit is limited to first 52 or 54 values, as digits cannot start an identifier). Each character then takes exactly 6 bits, so one can fit four of these in a 24-bit word, or five in a 30-bit word.This kind ofperfect hashingis useful, because it is bijective. Instead of storing the script as text, one can tokenize it as it is read, and convert tokens back to string if a listing is desired. This is what old BASIC versions usually did, to save memory.I personally like to use DJB2 Xor hash instead, because it is trivial, but has few collisions among ASCII alphanumeric strings. It does mean that there might be more than one string corresponding to each identifier, so I recommend against using it for cases when the user can add new identifiers (like variables).When you have a microcontroller that reads in new values for state variables from an UART, then the UART receive interrupt can be a simple state machine, that in the state variable name phase just hashes the name; and in the assignment or lookup phase, does a binary search into a sorted array of hashes, with each hash corresponding to a type and a pointer in RAM. This naturally leads to a command mini-language where each identifier is followed by the command character (=for assignment,?for query), followed by optional value, followed by end of command character (usually newline). The array of hashes+types+RAM-pointers can reside in ROM/Flash, and for N entries, 3+⌈log2N⌉ lookups are enough to locate the correct entry, if any.With GCC and Clang, one can use the section attributes to collect such entries from each compilation unit to a single section at link time. One then does need to do a custom fixup pass, extracting the section from the target object file, sorting the entries for binary searching, and write the section back. Extraction and reinsertion is trivial withobjcopy(which is part of the development tools used with GCC or Clang), but you do need to sort the binary data (extracted into a separate file, containing only the binary data) yourself using a helper script or program. Such a fixup pass is also useful for detecting if identifier hashes collide; the sorter does not need to even know the hash algorithm used.

author: brucehoult
author_rank: Super Contributor
timestamp: October 11, 2022, 12:35:20 am
content: Quote from: Nominal Animal on October 10, 2022, 10:54:29 pmReinventing DEC's RAD50?There is a reason 6.3 was a popular format for file names.

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 11, 2022, 01:44:32 am
content: Quote from: brucehoult on October 11, 2022, 12:35:20 amNot reinventing, just describing!The key in my post was that if one has a perfect hash function, or a hash function with only acceptable collisions, then a microcontroller firmware that accepts assignments and responds to queries of named variables does not need an input buffer at all; and that encoding schemes likeDEC RADIX-50, and the inverses ofBase64andBase85, can be considered perfect hash functions. Interpreters can use the perfect (bijective) hashes of the keywords as tokens, too.One variant of Base32 I've used consists of uniquely identificable letters and digits: 0OQDo, 1Il, 2Zz, 3, 4A, 5Ss, 6Gb, 7, 8B, 9gq, a, Cc, Dd, Ee, Ff, Hh, i, Jj, Kk, L, Mm, Nn, Pp, Rr, Tt, Uu, Vv, Ww, Xx, Yy, dot or decimal point, and leading blank or non-leading underscore. This uses 5 bits per character, so a 6-character identifier fits in 30 bits nicely. In a microcontroller, you use a 128-byte array in ROM/Flash to map incoming characters to identifiers, and if you keep only the last 6 characters, you just shift the current hash left by 5 bits and add the next identifier character in the identifier parsing stage. (If you want to keep only the 6 first characters, only add the current identifier character if the current hash is less than 2²⁵.)I used it for identifiers like VOL.1, with the express intent that typos like v0l.I and VQI.l are acceptable forms for the same identifier. Whether you consider it a perfect hash function or an imperfect one is up to discussion, but I consider it perfect hashing with multiple acceptable input forms.If I recall correctly, HP originally used something like this with 7+ -segment LED displays: so, I definitely didn't invent this either!

author: peter-h
author_rank: Super Contributor
timestamp: October 11, 2022, 11:23:56 am
content: In 1991 I developed a product which had a Tiny Basic. A few k of code, Z80 asm. Integer only, but usable for byte processing. The same product also has a fairly full Pascal compiler, written in Z80 asm ~1975 and licensed to me for that product; most people ended up using that since it was far faster and better. It took up ~20k. I also implemented a Wordstar compatible editor. All this runs with a VT100 terminalAnd 30 years later it still sells. Most were actually programmed in C, using a compiler on a PC.You should be able to find a full Basic in C. More or less everybody can use that straight away.AFAIK Python (C source) is about 50k of ARM32 code and needs another 50k of RAM. Some love Python and the rest don't want itThe Q to ask is what is the value of being able to plug in a dumb terminal (there is no such thing; you would have to use a laptop running Teraterm or some such) and write/edit source code in the target. It has a certain intellectual appeal but nowadays it is limited.

author: brucehoult
author_rank: Super Contributor
timestamp: October 11, 2022, 03:34:58 pm
content: Quote from: peter-h on October 11, 2022, 11:23:56 amYou don't need a PC.This $36.90 device has a 320x240 colour display (24x40 of 7x9 font with 1 pixel gaps), 192 KB RAM, 4 MB flash, 120 MHz Cortex 4MF, WIFI and Bluetooth, SD card, 40 pin GPIO (Pi compatible), SPI, I2C, I2S, ADC, DAC, PWM, UART, 5-way switch, 3 buttons, accelerometer, light sensor, IR transmitter, microphone, speaker. All in a nice case.Interface it to a BT keyboard and load appropriate software and it's a literal terminal.https://www.seeedstudio.com/Wio-Terminal-p-4509.html

author: newbrain
author_rank: Super Contributor
timestamp: October 11, 2022, 03:45:56 pm
content: Quote from: brucehoult on October 11, 2022, 03:34:58 pmJust a magnifying glass, at 2.4"But apart from the use as a terminal, interesting dev board.

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 11, 2022, 06:47:46 pm
content: One can always take one of the Linux single-board computers like Orange Pi Lite, compile a monolithic kernel that does not require an initrd to boot, and replace the userspace with a dedicated terminal application. For eye candy, compile Qt5 for framebuffer and OpenGL ES 2.0, and you get full resolution accelerated graphics with antialiasing, including nice fast font rendering. This way, you can get boot times down to seconds on many SBCs.I've got a couple of Linux minilaptops (Acer Aspire One A110L and Asus Eee PC) that I might convert to that use. Any netbook or laptop with sufficient Linux support works fine for this. They could be fast enough for use as X11 terminals over Ethernet, too, although their resolution is typically 1024×600 to 1366×768, maybe down to 800×480 on the Eee PC; I'd need to check the version. The BIOS part of their boot sequence is quite slow, though.The userspace integration work begins at the init system. Systemd is slow and resource-hungry, and not needed in appliance use. You can write an init script that starts needed services in sequence where necessary, and the rest in parallel, to minimize boot time. (If you want Ethernet or WiFi, or device detection on plug-in, you do need a few service daemons running.) It's interesting stuff, too; perfect for someone interested in embedded appliance development as a proof of concept or portfolio example.

author: brucehoult
author_rank: Super Contributor
timestamp: October 11, 2022, 09:15:50 pm
content: Quote from: newbrain on October 11, 2022, 03:45:56 pm167 PPI. That's the same as my 17" MacBookPro (pre "retina") on which I run 9 pt Monaco / ProFont / Ubuntu Mono no problems.

author: newbrain
author_rank: Super Contributor
timestamp: October 11, 2022, 10:04:32 pm
content: Quote from: brucehoult on October 11, 2022, 09:15:50 pmSo you are telling me that a pixel drawn 9/167x25.4 = 1.37 mm height (and that's for the tallest letters, the x height is about half!) font is legible in an useful way.To this I there's only one reasonable answer: I'm green with envy for your eyesight.AFAIK, the 17" Macbook Pro has a 1920x1200 resolution, that puts it more in the 133 ppi range; in any case, a correctly rendered 9pointsfont will be about 3.2 mm independent of screen resolution.A 9pixelsfont will instead be 1.72 mm, still awfully small, but 25% larger, and overall better especially considering the difference in quality of the panels.The original "bitmapped 9pt" was designed for the original Mac, that had about 72 ppi, i.e. 1 pixel = 1 point.

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 11, 2022, 10:56:44 pm
content: Quote from: brucehoult on October 11, 2022, 09:15:50 pmSure, but 9pt is 21 pixels tall at 167 PPI (1 pt = 1"/72 = 0.352778 mm).At 167 PPI, an 8px × 9px font is 1.22mm×1.37mm or 3.9 pt, and 8 px × 10 px font is 1.22mm×1.53mm or 4.3 pt. These give you a 40×26 or 40×24 character display on a 320×240 pixel display module.I do believe the smallest size considered legible in newspapers is one Agate, or 5.5 pt, or about 1"/13 ≃ 1.95mm. (This size being the full line height, just like when considering fixed-width or variable-width but fixed-height bitmap fonts.)

author: brucehoult
author_rank: Super Contributor
timestamp: October 11, 2022, 11:40:30 pm
content: Quote from: newbrain on October 11, 2022, 10:04:32 pmI don't have one, so I don't know how good the screen is.QuoteAh dammit you're right. And the 4k (3840x1920) 32" monitors I use on my desktop machines are 138 PPI. I was going from memory. Now I'm wondering what it is that I've used that was ~160 PPI.QuoteThis is true, but that is a terminology argument that was lost 35 years ago. The font is called Monaco 9 and is 9 pixels high no matter what the screen resolution. It is 9 pt high on the original Mac's rather low resolution 72 DPI screen.QuoteAt 133 or 138 PPI 9 pixel high characters are fine for things that get referred to occasionally, for example "top", especially on the MBPs very good screen, though it's ok on the cheap Philips 32" 4K. For things I'm spending all day looking at (my source code) I bump it up to 11 on the MBP and 12 on the 4K.QuoteUh, I dunno. I don't think it's anything special. I'll be 60 later this year and have had astigmatism since childhood, though it wasn't diagnosed until I was 35 and my eye muscles started to get fatigued after a few hours of distorting my eyeball to correct it, so I got glasses and all was fine again.When I was a kid I only knew people would talk about details of far-away objects and I'd struggle to even see the objects.I renewed my truck license a couple of years ago, and I'm still legally ok to drive heavy vehicles without glasses. Or fly a private aircraft.Last time I got new glasses they did a fresh prescription, ordered the lenses/glasses, and when I got them I immediately said "they're wrong -- they're worse than my old glasses". They measured the lenses and they matched the prescription as measured a couple of weeks earlier. They told me to take them home and use them for a few days to get used to them. I did so, and came back. I said "no ... I can't use these ... the left lens needs to be rotated to the left 5 degrees and the right lens needs to be rotated to the right 10 degrees". Which I determined by holding the glasses in front of me and rotating them.They said "We only measure the prescription to 15 degrees!". "Well, I'm not going to use these ones. I can tell the 10 year old glasses are not quite correct now, but they're better than these". They re-ordered with my corrections and the glasses came back great.sigh. And that was a decade ago, so I'm past due to go through that again.Having now got the correct PPI for my computer screen I'm prepared to concede that a 9 pixel font is probably too small for even casual use on a 2.4" 320x240 screen :-)

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 12, 2022, 03:55:50 am
content: Quote from: brucehoult on October 11, 2022, 11:40:30 pmWell, you could have said that, then: 9px Monaco 9, instead of 9pt Monaco!Monaco itselfis a TrueType font nowadays, with the bitmap font (Monaco 9) a historic specialty in macOS.Quote from: brucehoult on October 11, 2022, 11:40:30 pmRighty-o. It's sufficiently readable for rare debugging and such, but not for casual use – especially if it is a TN panel with limited viewing angles.(One of my forever projects is adding an 2.8" 240×320 IPS display to a MikroTik RBM33G with a 4G/LTE modem/edge firewall for rural use, to display connection status in an user-friendly manner (when a button/capacitive sensor is triggered), but also to display the IP address or error message when needed for remote debugging. I have a very similar problem: the display is too small to display lots of stuff, so I'm going for user-friendly animated graphics instead. But to not look utterly crappy, that needs a fast display update rate, which is borderline with the typical SPI connections to a ILI9341 controller, so I'm moving to parallel 8080-style bus comms. Which has its own interesting problems, not the least of which is the parallel bus pinouts from various MCUs I have. Forever project...)

author: nctnico
author_rank: Super Contributor
timestamp: October 12, 2022, 09:07:20 pm
content: Quote from: Picuino on October 09, 2022, 04:56:22 pmAfter reading the thread I think what you need is a command line interface that can execute commands that are send over some kind of communication interface (serial port is the easiest). Not a scripting language. A scripting language on a microcontroller is only useful to make easy (in field) changes to the high level functionality of a device.Edit: a command line interface is easy to adapt to a simple form of scripting. The u-boot bootloader is an example.

author: Picuino
author_rank: Super Contributor
timestamp: October 13, 2022, 07:43:51 am
content: Yes, but a simple command line interface is not enough. I want it to be programmable as well. What I like the most so far is theTethered Forth. It serves as a command line and programming language all in one, with very little weight inside the microcontroller. I'm going to try to develop something similar for Arduino.

author: nctnico
author_rank: Super Contributor
timestamp: October 13, 2022, 11:56:22 am
content: Take bash as another example for scripting a command line. Using an archaic language is not going to be very popular amongst people that need to work on your project

author: SiliconWizard
author_rank: Super Contributor
timestamp: October 13, 2022, 05:31:53 pm
content: Is the goal to actually implement projects with this, or just to offer a new "scripting" language for MCUs (which the added "something similar for Arduino" kinda suggested, but maybe I'm reading a bit too much into it.) Just curious.

author: Picuino
author_rank: Super Contributor
timestamp: October 13, 2022, 06:06:35 pm
content: The objective is to control the microcontroller from the PC, both in command line and with automatic macros.Forth does not have to be the language of choice. In fact I am thinking of doing the same, but with Python as the PC side language. It would be a Tethered-Python.On the microcontroller side, an interrupt will capture the commands sent by the PC and execute the function corresponding to the command sent.The microcontroller side program I want to program it in C (not C++) so that it is portable from the Arduino to other microcontrollers like the PICs that I also use.

author: Sal Ammoniac
author_rank: Super Contributor
timestamp: October 13, 2022, 08:09:34 pm
content: When I had a similar need a few years ago, I implemented a small Forth in ARM assembly language. The core was ~1000 lines and the rest was Forth source code.The whole thing works fine--the only problem with it is that IhateForth!

author: SiliconWizard
author_rank: Super Contributor
timestamp: October 13, 2022, 08:15:46 pm
content: Forth is probably hard to beat in terms of code footprint. Micropython and Circuitpython are becoming popular, although they will certainly require more memory than Forth.

author: ogden
author_rank: Super Contributor
timestamp: October 13, 2022, 08:34:43 pm
content: Unless you are fighting to shave every cent off the final product price which you dream to sell in >=5 figure quantities, code footprint of interpreter shall not be concern. Hint: cost of STM32F411-based (512KB flash) "black pill" board is around 10$.

author: Picuino
author_rank: Super Contributor
timestamp: October 14, 2022, 09:57:37 am
content: I usually use 8-bit microcontrollers because they still use 5 Volt outputs. Some 16-bit PIC microcontrollers also use 5 Volts, but they tend to have little Flash memory as well.It is true that more and more peripherals and circuits are running on 3.3V, but there are still many 5V ones.

author: DiTBho
author_rank: Super Contributor
timestamp: October 14, 2022, 10:33:12 am
content: 1024x768 is more than enoughFor me

author: ogden
author_rank: Super Contributor
timestamp: October 14, 2022, 07:42:54 pm
content: Quote from: Picuino on October 14, 2022, 09:57:37 amYou rarely get it all. 5V I/O, tiny MCU with loads of memory and performance for decent interpreter does not exist. I would suggest to reconsider 5V I/O (of last century) requirement, think of level shifters if needed but pick modern materials, tools and solutions.

author: SiliconWizard
author_rank: Super Contributor
timestamp: October 14, 2022, 08:15:36 pm
content: Yeah. Many 5V stuff? Sure. Uh... Not really. But as always, if your main tool is a hammer, all you'll see everywhere will be nails.

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 14, 2022, 10:46:50 pm
content: Is there an intended difference between "command languages" and "programming languages"? Or are we really talking about query-response interfaces?What are the practical use cases here?For example:Let's say that we have an input lexer-parser that converts incoming text into tokens stored in a list (or first-in, first-out stack). Each token is either a known name, or a numerical value, and uses the exact same amount of memory (say, 32 bits). Whenever the input is complete (for example, a newline is received), a handler function is called to handle the token list, perhaps selecting the exact function based on the initial token.This is not a "language" per se, but a query-response interface. If you have a perfect hash function (as I discussed earlier), the input parser does not need to keep a copy of the input string at all. It can be a relatively simple state machine that consumes the input, and poops out a list of tokens. The RAM use is minimal.Theproperty=valueandproperty?interface I described earlier, parses initially the property name, with the following character choosing which function will eventually be called. If the character is=, then the following numerical value is parsed. Otherwise, there should be nothing on the line except whitespace (or comments) before the newline. This needs only a few bytes of RAM to implement, with almost everything in ROM/Flash, which is very useful for memory-constrained tiny microcontrollers.The more shell-like form (space-delimited words as tokens), as discussed briefly before, with e.g.setpropertyvalueandgetproperty, is not much more complicated, but does need RAM for the token list. Its state machine can actually be even simpler. It is a lot more versatile while still having a very intuitive action/command interface as well (command[arguments...).Variables are easiest to implement if they always start with a sigil, say$, used also when assigned to (unlike in Bourne or POSIX shells). Then, each token is either a known name, a numerical value, or references a variable; with variable values stored in a dictionary –– basically a set of token-value or token-token pairs. When the hardware supports 32-bit floating-point (IEEE 754 Binary32, aka "float" in most situations nowadays), one can use NaNs (over four million distinct values) for non-numeric tokens. This is very useful, because that way all tokens are exactly 32 bits.In all cases, for known names, one can use a hash table in Flash/ROM for very quick lookup and no RAM used.However, with this approach, there is nocode listing, just individual commands and assignments and queries, that are acted upon when completely received, and then discarded. You can implement short loops and repeats (as long as they fit in a single command, that is), and even conditionals and aliases, but procedures or functions would require separating the execution from parsing the tokens: you'd need to store the list of token lists, and execute them separately. (A simple option is to use a separator token, like;in POSIX and Bourne shells, but the big difference is separating the execution from the parser. Technically, it is not difficult, but depending on the implementation, can be quite "tricky". It is better to design the system for the intended model from the ground up.)So, to get the discussion forward, what exactly do we need from the interface we're talking about?I do not see myself needing any kind of full language with function support in a tiny microcontroller at all. However, the command/query-response interfaces I describe hereareuseful, and definitely nothing new. And they can be made with very frugal resource use, too. Minimal code is needed to access and control any (numerical) variable in RAM. Simple wrapper functions are needed to make commands that call firmware functions. My uses vary from experimentation to debugging. Granted, I do hobby stuff and a lot of personal experiments, not commercial products, so perhaps my own experience is flawed.(I also like full duplex more than a strict query-response interface, so that even the microcontroller can output similar format statements at any point in time (useful for timed stuff). That is, responses not necessarily being in the order of the queries, but in the order of completion, with additional events possibly reported in between. This means that I also like the way e.g. Gcode lets one specify an identifier for each command or query, with that identifier included in the response.Looking at "languages" likeG-codeandHPGLis useful, because they were (and still are) parsed and processed by 8-bit microcontrollers. Some of their features are historical baggage, but some are a careful balance between human readability and ease of machine processing.

author: Picuino
author_rank: Super Contributor
timestamp: October 15, 2022, 10:13:06 am
content: At first I was looking for a language similar to BASIC or FORTH so that the microcontroller could interpret it and give me control over its processes.Now I have realized that a language interpreted by the microcontroller takes up too much space, must be too simple and makes no sense.Since the interpreters on the microprocessor are hardly going to execute instructions faster than data coming in over a UART or SPI channel, it is better to move the programming language to the PC and send codes to the microcontroller that will be quickly executed.It is not even necessary to implement loops or conditionals. That can still be done by the PC. The only thing really necessary is to implement instructions in the microcontroller and a dispatcher that, depending on the arrival of one code or another, calls the different functions.For example the PC sends the code 0x20 and the microcontroller reads the analog input 1 and returns the value readed.The PC sends code 0x40 and the microcontroller sets digital output 1 to high level.Etc, etc.What I now need to know is how to optimize and standardize the codes sent to the microcontroller so that I don't have to rewrite the PC and microcontroller programs every time I make a new application.

author: tggzzz
author_rank: Super Contributor
timestamp: October 15, 2022, 10:54:10 am
content: You are using the microcontroller to control something ultimately using reads and write operations, with some additional processing to determine the specific reads and writes.What is the maximum allowablelatencybetween one such control action and the next?What is the maximumlatencyfor comms to and from the microcontroller and a PC application?Have you considered that a very fast edit-compile-download cycle would be as effective debugging environment as an interpreter?... especially if there is an additional REPL loop in the microcontroller for human-speed exploration of what's happening?That would enable you to:have compact and low latency simple read/writes/processing in the form of compiled functions running in the MCUexplore the next "higher" level ofcompositeactions by typing at the keyboardwhen that next level is sufficiently understood (i.e. debugged), compiling that and downloadind itRinse and repeat as you increase the complexity/sophistication of your actions.I have no specific environment in mind, but I would be surprised if you couldn't find a Forth that could do it.

author: DiTBho
author_rank: Super Contributor
timestamp: October 15, 2022, 11:05:09 am
content: Quote from: SiliconWizard on October 13, 2022, 08:15:46 pmStill problematic on my Casio SH3-hacked calculator.Too fat!!!

author: IDEngineer
author_rank: Super Contributor
timestamp: October 15, 2022, 02:59:01 pm
content: Quote from: Picuino on October 09, 2022, 03:51:32 pmI know I'm late to this thread, but OMG... we went through this exercise back in the 80's. "Compiled languages are too much, let's use an interpreter." "It will be faster, smaller, easier to learn, etc." We wasted man-months on this. In the end, it was just easier to use C for sooooo many reasons. It's an, um, "interesting" exercise and it's your time to spend, but having gone down this road I would never ever recommend it to anyone.

author: nctnico
author_rank: Super Contributor
timestamp: October 15, 2022, 04:03:35 pm
content: But back in the 80's Lua and Python did not exist and cpu power was limited. Nowadays a $1 ARM microcontroller runs circles around any PC from the 80's. On top of that scripting languages have huge ecosystems as well. IOW you can't compare like you do.

author: westfw
author_rank: Super Contributor
timestamp: October 15, 2022, 08:20:08 pm
content: QuoteWe should decide what we mean by "tiny microcontrollers."Lua and Python are pretty big by many standards (does Lua even run on smallish ARM chips?)No one has mentioned PCode, or the pre-tokenized BASICs.

author: IDEngineer
author_rank: Super Contributor
timestamp: October 15, 2022, 08:49:13 pm
content: Quote from: nctnico on October 15, 2022, 04:03:35 pmI was speaking more about the human factors, not the hardware. It turned out to be easier to coordinate the use of traditional compiled languages across a staff of Engineers.We tried the same thing for writing test code... same result.As with most things, the humans are usually the limiting factor.

author: NorthGuy
author_rank: Super Contributor
timestamp: October 15, 2022, 09:12:11 pm
content: IMHO, interpreted languages is pure bloat with no benefits.Many people seem to be obsessed with languages thinking that a particular choice of language makes them more productive or similar non-sense. This reminds me of people on the golf course who buy new modern drivers every year and swear they then can drive longer and play better because of their new driver, but if you look at their scores, they're still the same.

author: tggzzz
author_rank: Super Contributor
timestamp: October 15, 2022, 10:39:20 pm
content: Quote from: NorthGuy on October 15, 2022, 09:12:11 pmAll languages are interpreted, but the level varies.For example, Pascal pCode was interpreted very successfully on 8080 class machines.A current example: if you think assembler code (and hence C/C++/etc) isn't interpreted on moderm x86-64 processors, then you don't understand the implementation of the processors. Each generation of processor has a significantly different interpreted, the details of which are not released by intel. The ability to change the interpreter is how intel fixes bugs on installed x86-64 processors.The ability to have a fast interactive mechanism for "trying out" code is key to productivity. You cannot appreciate that until you have experienced it. That can be a conventional interpreted language or some modern compiled languages (not C++, not Fortran, not C). The lack of facilities in many languages lead to "Greenspun's Tenth Rule of Programming: any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp."But I agree with you to the exent that the specific language is unimportant.

author: PlainName
author_rank: Super Contributor
timestamp: October 15, 2022, 11:02:04 pm
content: Quote from: IDEngineer on October 15, 2022, 02:59:01 pmI am kind of following this thread because it might be of interest. As an example use, I am thinking of our garage light controller which is currently a C application that reads a config file to figure out how to control the lights. To change things you could just upload a new config file. But... sometimes I want to try something, or have something additional occur, temporarily. Maybe it's a special circumstance, or I'm just interested in how often a particular event occurs, or maybe I want to know the temperature (each node has a thermometer and the protocol for accessing them is defined but currently not used).It would be cool to fire up the web browser and type some script into an edit box and run it. Or remove a previously entered one. Don't really want to get out the compiler and libs and everything and do an OTA upgrade just to check something, then an OTA downgrade...Quote from: PicuinoNever used Forth (the one time I came across it, a 40x20 or similar editing screen was enough to put me off), but I could learn to live with it. However, TF is written in assembler, and I think something that might be ported should be written in C since not every destination will be an ARM-based MCH. Portable Forth is apparently ANSI C, but I don't know whether it would be small enough.I like the idea of and onboard language, over just a bunch of functions a PC could call via a comms link, because it is standalone. Unless you intend to make an accessory for a PC, you would want to be able to tell it to do stuff and have it keep doing that.

author: NorthGuy
author_rank: Super Contributor
timestamp: October 15, 2022, 11:07:02 pm
content: Quote from: tggzzz on October 15, 2022, 10:39:20 pmSure. I do this all the time. Compiling a small snippet of C or Assembler code and programming it on the chip takes couple seconds. How much faster do you need?

author: PlainName
author_rank: Super Contributor
timestamp: October 15, 2022, 11:26:54 pm
content: QuoteI realise now that I am typing that, and looking at it, as an end user whereas most of you seem to be seeing it from the viewpoint of a developer.

author: brucehoult
author_rank: Super Contributor
timestamp: October 15, 2022, 11:43:25 pm
content: Quote from: westfw on October 15, 2022, 08:20:08 pmFor me that certainly means 8051, PIC, AVR, maybe MSP430.All but MSP430 are Harvard architecture so you don't even have the possibility to JIT new code into RAM. not even simple subroutine threading. So you're limited to token or address threading at best, both of which have quite high overheads on ... all but MSP430.MSP430 is sufficiently PDP11-like that it can use any register (let's say R4) as pseudoPC and with a single 16 bit instruction fetch an address-threaded word from the address pointed to by that register, increment the register, and jump to or call native code at that address.So native code functions called from threaded code end withMOV @pseudoPC+,PC, which calls another native code function. If the thing being called is actually threaded code then it starts withCALL threaded_entrywhich can be a single 16 bit word if the address ofthreaded_entryis always kept in a register e.g.CALL R5, followed by a list of addresses of functions to be called. ThethreadedEntryneeds to save the old value of the pseudoPC on a stack, pop the return address (which is really the address of threaded code pointers) from theCALLinto pseudo-PC, then call the function that address:Code:threadedEntry:MOV @SP+,tmpPUSH pseudoPCMOV tmp,pseudoPCMOV @pseudoPC+,PCAnd, finally, threaded code has to end with a pointer to theRETURNfunction that pops a saved pseudoPC from the stack and calls the threaded-address function it points to.Code:RETURN:MOV @SP+,pseudoPCMOV @pseudoPC+,PCI think I got that about right -- it's been nearly 40 years since I thought about this stuff.So, a threaded subroutine that consists of a series of calls to native subroutines executes with just ONE INSTRUCTION overhead (and two memory reads, one for the opcode, one for the address to call) between each native code function. Space-wise it one 16 bit word for each function called (whether native or threaded), plus one word native code prolog and one word for the call to RETURN.Calling a threaded subroutine has 5 instructions overhead on entry and 3 on return.All of 8051, AVR, PIC will have much much worse speed penalty. Or 6502 and z80 for that matter.QuoteTended to need 8 or 16 KB ROM plus 32+ KB RAM to be useful. UCSD Pascal needed 64k RAM, and even then every time you moved between Filer / Editor / Compiler it swapped probably 30 KB of code out and in at 30 KB/sec on the floppy drive(s) ... on the Apple ][ in 1981 it literally went ka CHUNK dit dit dit dit dit dit dit CHUNK dit dit dit dit dit dit dit and took a couple of seconds. (it was 4 KB per track, 40 tracks per disk)

author: tggzzz
author_rank: Super Contributor
timestamp: October 15, 2022, 11:49:11 pm
content: Quote from: NorthGuy on October 15, 2022, 11:07:02 pmAfter a while it will be more than a small snippet or couple of lines of C, and itwilltake more than a couple of seconds to compile and download. Even 1 minute is enough to break the flow of interactive debugging. Whether breaking the flow is a problem is a separate issue, of course.The ability to instantly try something without noticeable time penalty (compilation or execution) is valuable. You have to have experienced it to realise that.

author: tggzzz
author_rank: Super Contributor
timestamp: October 16, 2022, 12:08:10 am
content: Quote from: brucehoult on October 15, 2022, 11:43:25 pmThat was positively speedy compared with Apple's Smalltalk-80 running on a Fat Mac"Glacial" is the appropriate word. Nonetheless it was sufficient (in 1986) to make me realise the advantages of OOP.

author: brucehoult
author_rank: Super Contributor
timestamp: October 16, 2022, 12:15:07 am
content: Quote from: tggzzz on October 15, 2022, 10:39:20 pmYou needed 64 KB RAM (or RAM+ROM if you put the P-Code interpreter into ROM) on 8080/z80/6502 to run a useful P-Code system.That's a BIG MACHINE in today's microcontroller landscape. There are still many many things done on machines with 512 bytes, 2k, 8k, 16k of RAM. Sure, they often have much more ROM (flash) than that, but that's usable by compiled and downloaded C, not by on-the-fly functions downloaded as text and interpreted as text, tokens, or addresses.Quotex86 is an abomination.RISC-V code as seen in microcontrollers (e.g. RV32IC) is NOT interpreted. It is directly executed by the hardware. The 16 or 32 bit opcodes are combinatorially decoded into something like 60 bits of signals (including 15 bits for two source and one destination register, 32 bits for a constant/offset, 4 bits for ALU operation, and misc signals such as is_jmp, is_write etc) which then DIRECTLY CONTROL the hardware gates.QuoteAbsolutely true.QuoteIt absolutely can be.I just checked with my HiFive1 board. After editing the program I was working on most recently, the times are:0.7 seconds to compile 28 C/C++ files and link to an ELF file0.5 seconds to flash the new code to the board via USBWould 0.1 seconds total be better? Sure, but not much. I generally spend a LOT more than 1 second doing even something as simple as changing a constant or adding a printf() somewhere.QuoteAs someone who has written a LOT of code in Dylan (roughly: Scheme semantics, Common Lisp library including CLOS, and Wirthian syntax), quite a lot in Scheme, and a reasonable amount in Common Lisp I understand where you are coming from and have known this quote for several decades.Python, Ruby, and JavaScript are effectively the same as "Common Lisp" as far as that quote goes. Modern compiled languages such as Scala and Julia too.

author: ogden
author_rank: Super Contributor
timestamp: October 16, 2022, 12:15:56 am
content: Quote from: dunkemhigh on October 15, 2022, 11:26:54 pmBoth user and developer may be interested in scripting language - in case of highly customizable product. User would want to avoid hidden costs which can come if customization will be done by developer, on the other hand developer would want to avoid developing complex customization system/GUI because it may add to cost, up-to rendering whole project non-feasible for user. Solution: add embedded scripting language.

author: brucehoult
author_rank: Super Contributor
timestamp: October 16, 2022, 12:31:37 am
content: Quote from: brucehoult on October 15, 2022, 11:43:25 pmIMPORTANT NOTE: the above threaded code mechanism has NOTHING to do with FORTH.It is true that FORTH is often implemented in this way, but so have been many compilers for FORTRAN, COBOL, Pascal and probably even C or BCPL or Algol on machines with limited memory size and weak instruction sets leading to bulky native code.

author: NorthGuy
author_rank: Super Contributor
timestamp: October 16, 2022, 12:39:57 am
content: Quote from: tggzzz on October 15, 2022, 11:49:11 pmTo create something that big you will need months of development. This is not something you can possibly try quickly, and it will take really long time to duplicate with interactive interpreted language.

author: brucehoult
author_rank: Super Contributor
timestamp: October 16, 2022, 01:45:10 am
content: Quote from: NorthGuy on October 16, 2022, 12:39:57 amomg. "Even 1 minute".Yes, I used to have those kinds of turn-around times, back in the 80s. Hell, I remember back when I first started using Apple's "MacApp" application framework clean build times were about an hour. It was an amazing day when I got a 60 MHz PowerPC Mac with the new Metroworks CodeWarrior and suddenly those clean build times were under five minutes.I just tried modifying one of my HiFive1 programs to include a 1 MB static array of data, initialised with 2^20 entries like0xb7,, but random.2.5 sec compile time (up from 0.7 seconds for a small program)7.5 sec upload/flash time (up from 0.5 seconds for a small program)What would it take to get to a minute? SIX megabytes of program?Yeah, that board has that much flash, and more. But only 16 KB RAM. What would you even DO with 6 MB of program?Apart from this, back in February 2017:I got a copyright claim on that video, lol.

author: brucehoult
author_rank: Super Contributor
timestamp: October 16, 2022, 02:34:30 am
content: Quote from: tggzzz on October 16, 2022, 12:08:10 amOMG! Someone made a video earlier this year of doing some programming on an Apple //e using Apple (USCD) Pascal.Sooooooo slow. 1 MHz 6502 forever. The UCSD P-system was a port from other machines -- exactly the same bytecode programs ran on the PDP-11, 8080/z80, M68000, and 8086 -- and really wasn't designed or optimised around the 6502's strengths and weaknesses. I think quite a bit better could be done.Basically the exact system you see here was one of the original three OSes offered for the IBM PC, along with CP/M and MS-DOS. Hopefully it was a bit faster.I don't think the 128k made any real difference from 64k in this video. The compiler is getting down to 34 KB free, which obviously could not be 64 KB lower on a 64 KB machine. Maybe it is keeping more code overlays in RAM at the same time so repeated flipping between editor and compiler is faster.

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 16, 2022, 03:21:28 am
content: The machinery I use (with GCC or Clang, since these require the toolchain to use ELF format object files), means that if I want to expose saystatic int beef = 1;to my query-response interface, all I need to do is include a header file defining some structures and macros, and addEXPOSE_INT(&beef,0xBEEF);where0xBEEFis the token identifier (perfect hash of the name) I use in the query-response interface for it.That, and an input parser/tokeniser and dispatcher, is all that is needed, really.The key parts of that included header file areCode:#include <stdint.h>struct exposed {uint32_t ident;union {void (*func)(void);int *iptr;/* Other pointer types omitted for brevity */};} __attribute__((packed));#define MERGE2_(a, b) a ## b#define MERGE2(a, b) MERGE2_(a, b)#define EXPOSE_(varname, vtype, vname, member, ptr) \__attribute__((used, section ("exposed"))) \static const struct exposed varname = { \.ident = (((uint32_t)(vtype) & 0x1F) << 27) \| ((uint32_t)(vname) & 0x07FFFFFF), \.member = (ptr) \}#define EXPOSE_FUNC(fn, hash) \EXPOSE_(MERGE2(expose_, __LINE__), 0, hash, func, fn)#define EXPOSE_INT(ptr, hash) \EXPOSE_(MERGE2(expose_, __LINE__), 1, hash, iptr, ptr)/* Up to 30 other types omitted for brevity */The union is only used for type checking, so that if you use e.gchar step;and thenEXPOSE_INT(&step, 0xDEAD), you get a warning about incompatible pointer types.Above, the high 5 bits ofidentspecify the type, and low 27 bits the perfect hash or encoding of the name. (For example, five base-40 characters.)EachEXPOSE_INT()orEXPOSE_FUNC()declaration adds astruct exposedentry (in pseudo-random order) to theexposedsection in the final binary, where they all can be accessed. This means that I can expose evenstaticvariables, without global linkage, and it only needs a single declaration in that same compilation unit. It also works very well if one uses preprocessor conditionals or Makefile magic to select what code gets included in the final binary. Wherever the variable or function is defined, you just add an expose macro next to it: very low cognitive overhead.In the linked binary, we can use the following to look up any exposed item:Code:extern const struct exposed __start_exposed[];extern const struct exposed __stop_exposed[];const struct exposed *find_exposed(uint32_t hash){const uint32_t mask = 0x07FFFFFF;const struct exposed *curr = __start_exposed;/* Ensure hash has no type bits set. */hash &= mask;while (curr < __stop_exposed)if ((curr->ident & mask) == hash)return curr;elsecurr++;return NULL;}The two extern symbols are defined by the linker. The section contains all expose declarations from all compilation units linked to the final object.Iffind_exposed(hash)returns a non-NULL pointerp, then(p->ident>>27)identifies the type (the number in the EXPOSE_INT(), EXPOSE_FUNC() macros).All of the above gets placed in ROM/Flash; no additional RAM is used. (If you use your own linker script, this is theexposedsection. Do also define the__start_exposedand__stop_exposedsymbols, as is done by default linker scripts.)Thefind_exposed()operation can be made faster by adding a post-processing step in your build, where the order of the entries in theexposedsection is sorted (by the 27 low bits ofidentmembers, here). That would allow you to use a binary search, as I mentioned before. You could also store the strings, and in the post-processing step replace the strings with their corresponding hashes, even if it changes the size of theexposedsection. (The annoying thing is the relocation records, and editing them. Raw binary data is trivial, but relocation records –– the pointers to the variables or functions –– are harder to manipulate usingobjcopyonly.) (Also note that this latterstruct exposeddoes not need to have the same format as earlier; it only needs to match the possibly post-processedexposedsection contents.)In addition to above, you need the input parser, of course. But all it needs, really, is to parse (hash) the first name, look it up, and decide what to do with the rest of the input request/command –– but knowing the type and address of the thing being referred to. In particular, an implementation that supportsname'?'(query)name'=' VALUE(assignment)name(function call without parameters)nameVALUE(function call with an integer parameter)can parse the input character by character using a straightforward state machine, without any kind of string buffer at all.VALUEcan be an integer, fixed-point, or even a name, depending on what your exact needs are; but integer-only is the simplest. Adding multiple-parameter function calls requires a slightly more complicated state machine, and RAM storage for the parameters, a stack of VALUEs.This does come in useful, when you have an external device –– a sensor, or actuator of some kind, for example –– and you are not sure of the optimal parameters to use with that device. Or you are playing with e.g. PWM capabilities of the MCU.Instead of making everything externally programmable, you just declare the variables you want to get and set, and the functions you want to call, and your input parser stuff should handle the details.(As I often use Teensies for this kind of experimentation, and they all have native USB interfaces, I can also use an "extra" USB Serial endpoint for this kind of communication between my computer and the microcontrooler. The actual firmware can be a bulk USB device, a HID device, or whatever I happen to need.)In a design sense, this is very close to attaching a debugger to your microcontroller, and using it to modify the RAM state or execute specific functions.For playing with controlling external devices, I like this approach better than a debugger, because this way the microcontroller firmware runs normally and is always in full control, instead of being under the debugger control. This minimizes the differences between "normal operation" and "exploring settings", and thus ensures the results found via such exploration are consistent even when used with this machinery later on.

author: PlainName
author_rank: Super Contributor
timestamp: October 16, 2022, 08:28:17 am
content: QuoteThat's not an appropriate test since you are set up to do that. What would be the times if you start with just your source files, no cached headers or compiler output, no cached libs or framework?

author: tggzzz
author_rank: Super Contributor
timestamp: October 16, 2022, 09:01:05 am
content: Quote from: brucehoult on October 16, 2022, 01:45:10 amThat comment has to be read in conjunction with the last sentence in that paragraph!Slow turnaround times leads to a rather different style: you read, understand, think, and typing/compiling/running comes at the very endSometimes the old dayswerebetterQuote5minutes? Luxury, but then I started with Algol-60 on 5-channel paper tape and a 2kIPS machine (one is still running at TNMoC, and the staff are happy to whip out and discuss the schematics!)I always hated letter-shift figure-shift, only to find it has been trumphantly resurrected on the keyboards on fondle-slab computers and phones.QuoteNo closed-caption subtitles available

author: tggzzz
author_rank: Super Contributor
timestamp: October 16, 2022, 09:05:29 am
content: Quote from: NorthGuy on October 16, 2022, 12:39:57 amNow consider trying to debug an application with an included library. Too often the library is poorly documented and/or understood, and interactive step-wise debugging is a good way to proceed.

author: tggzzz
author_rank: Super Contributor
timestamp: October 16, 2022, 09:21:10 am
content: Quote from: brucehoult on October 16, 2022, 02:34:30 amDon't forget the Western Digitalhttps://en.wikipedia.org/wiki/Pascal_MicroEngineBut that p-code compilation is positively speedy compared with Apple Smalltalk on a FatMac. I'm tempted to make a video of it running; let's just say that even at 5fps, there could be a vast amount of inter-frame compression!

author: brucehoult
author_rank: Super Contributor
timestamp: October 16, 2022, 09:55:01 am
content: Quote from: dunkemhigh on October 16, 2022, 08:28:17 amThose are the times.Nothing is cached, there are no precompiled headers. All source files and headers are compiled from text every time.

author: brucehoult
author_rank: Super Contributor
timestamp: October 16, 2022, 10:17:26 am
content: Quote from: tggzzz on October 16, 2022, 09:21:10 amPlease do!It is hard to believe, given the Mac's CPU is approximately 10x faster. (estimate based on RAM bandwidth, typical instruction execution times, 2-4 times wider registers etc, not just the clock speed)THINK Pascal and Turbo Pascal both ran very well on 512k and 1 MB 8 MHz Macs, both the compile times and the executed code.Ah .. look and you will find. Here's a video using early Turbo Pascal on a Mac. It's a 16 MHz 68020, but on a 16 bit bus like the 68000. So it's probably not much more than twice as fast as the FatMac.Sadly the guy is a bit clueless. For example he says the SANE library imported is for the debugger. No. It's the Standard Apple Numeric Library (i.e. software IEEE FP). He was probably thinking of the SADE debugger (for putting your app in bondage) that came later with MPW (Macintosh Programmer's Workshop).He spends a long time going through the code. Actual compile and then run is at 33:00.

author: tggzzz
author_rank: Super Contributor
timestamp: October 16, 2022, 12:52:31 pm
content: Quote from: brucehoult on October 16, 2022, 10:17:26 amThe Apple Smalltalk was a "naive" implementation of the first Xerox image + VM.All accesses to all objects was indirect through the global object table. The bytecode was fully interpreted; there was no JITting (LP Deutsch's seminal JITter came later).Running from two floppies inserted into the single drive wasn't a significant limitation, due to the good consequences of the VM + image + incremental change file concepts.

author: tggzzz
author_rank: Super Contributor
timestamp: October 16, 2022, 02:37:12 pm
content: Quote from: tggzzz on October 15, 2022, 11:49:11 pmJust to emphasise that, here's something from a 39 year old book, indicating what hadalreadybeen achieved bfore 1983. It reminds me of the accurate old quip that "Algol-60 is an improvement on most of its successors".

author: SiliconWizard
author_rank: Super Contributor
timestamp: October 16, 2022, 06:35:53 pm
content: Quote from: NorthGuy on October 16, 2022, 12:39:57 amIt's a decades-old question that will keep popping up for decades to come.Some people seem to think that "interactive" development tools are more productive. My experience (and that of many others) is the exact opposite.

author: tggzzz
author_rank: Super Contributor
timestamp: October 16, 2022, 06:58:49 pm
content: Quote from: SiliconWizard on October 16, 2022, 06:35:53 pmIt depends.on the for circumstances.When I'm with people advocating interactive programming, I'll stress the benefits of planning and thinking before approaching a keyboard. OTOH when with people advocating compile-link-run then I'll stress the times when that is unnecessarily restrictive.Having many toolsand knowing when to use themis a key engineering skill. But you know that

author: IDEngineer
author_rank: Super Contributor
timestamp: October 16, 2022, 06:59:22 pm
content: Agreed. The best development environment / language / MCU / whatever is the one that gets the job done. Ask five Engineers for their preference and you'll get seven answers.WRT the cycle time discussion above, even with Microchip's MPLAB X and an ICD3 I can recompile and reflash in about five seconds. Takes me that long to spin in my chair from the firmware bench to the hardware bench and pick up the probe(s) to switch to physical debug mode. By the time the human is ready, the reflash is done and executing.I don't get wrapped around the axle about shaving sub-seconds off such cycle times, there are far too many other things that consume vastly more time and increase inefficiency to a greater extent. Great example: This year I built a "new" main workstation but could not go farther than an i7-something CPU and motherboard because I insist on running Win7. Many would argue that I'm "running slow" because I'm not using the latest i9-Thz CPU but let's be honest:* Most of the time the CPU is waiting on my next keystroke, so the human is already the limiting factor* Win7 is much easier to optimize for development by shedding all the garbage you don't need (I'm under 50 processes when idle, how 'bout you?)* Win7 never interrupts me demanding to perform an update, and never changes my settings to what Microsoft thinks is betterSo who is more efficient... the guy who can brag about a bleeding edge clock rate, or the guy whose machine isn't running unnecessary things and never interrupts for updates and changes no one asked for?

author: IDEngineer
author_rank: Super Contributor
timestamp: October 16, 2022, 06:59:53 pm
content: Quote from: tggzzz on October 16, 2022, 06:58:49 pmBest summary of this entire thread.

author: tggzzz
author_rank: Super Contributor
timestamp: October 16, 2022, 07:03:42 pm
content: Quote from: IDEngineer on October 16, 2022, 06:59:22 pm.Youshouldgetat leastseven answers

author: brucehoult
author_rank: Super Contributor
timestamp: October 16, 2022, 08:43:48 pm
content: Quote from: tggzzz on October 16, 2022, 02:37:12 pmThere are two kinds of programmers:1) those for whom 1983 is unimaginably far in the past, when dinosaurs roamed the Earth2) we dinosaurs

author: nctnico
author_rank: Super Contributor
timestamp: October 16, 2022, 08:52:28 pm
content: Quote from: tggzzz on October 15, 2022, 11:49:11 pmI agree. While developing software for microcontrollers I almost always implement commands to call certain functions (using a command line interface). That way I can try various combinations of parameters and/or redo the same tests later on for checking whether things still work as expected.

author: tggzzz
author_rank: Super Contributor
timestamp: October 16, 2022, 10:07:54 pm
content: Quote from: brucehoult on October 16, 2022, 08:43:48 pmI don't object to youngstersthinkingsome things are far in the past.I do object to them triumphantly reinventing an old concept, badly -and claiming it is new. I hate chrome plated elliptical wheels; they are too jarring.

author: tggzzz
author_rank: Super Contributor
timestamp: October 16, 2022, 10:13:31 pm
content: Quote from: nctnico on October 16, 2022, 08:52:28 pmWhen it comes to small memory MCUs, I hate stripping the fat out of scanf/printf over and over andoveragain. It is pleasant if someone has already done that for me within a tiny interpreter.And yes, in the 70s I did implement floating point arithmetic and trig amd i/o functions on a 6800; IIRC a sin/cos CORDIC pair took ~40ms. It was a vacation job after my first year at university. Very useful too, since another part of the task made me realise why/where sempahores are necessary.

author: IDEngineer
author_rank: Super Contributor
timestamp: October 16, 2022, 11:59:50 pm
content: I was speaking of asking about a specific development scenario. Even then you'll get multiple answers from single individuals. Heck, even I would probably have multiple answers. Might depend on which scope I feel like driving that day, which could end up making all other equipment choices.

author: NorthGuy
author_rank: Super Contributor
timestamp: October 17, 2022, 12:33:38 am
content: Quote from: IDEngineer on October 16, 2022, 06:59:22 pmMy main PC is i5-2500 running Windows 7. It's probably 10 years old but everything flies as I try to avoid bloated software, mostly, Vivado of course ...I used to upgrade PCs every 2-3 years, but then I didn't like Windows 8, and Windows 10 of course was a disaster where Microsoft is in full control. So, I wanted to go with Linux, but then I thought I'd stay with Windows 7. But it was too late. Now hardware gets more and more incompatible with Windows 7, so I keep procrastinating and deciding between Windows 7 (on not so new PC) and Linux. I have few PCs running Windows 8.1, Windows 10, and Windows 11, and I don't see anything in them more appealing compared to Windows 7.

author: brucehoult
author_rank: Super Contributor
timestamp: October 17, 2022, 03:36:35 am
content: You can always run Windows 7 in a VM on Linux for the few things that actually need Windows. VirtualBox works fine and is free. It's what I use to run corporate-mandated stuff.

author: PlainName
author_rank: Super Contributor
timestamp: October 17, 2022, 10:49:07 am
content: QuoteThere are some thing that need Windows, but for me it's as much about workflow as anything else. Does Linux have Listary or anything even remotely close, for instance? DOpus? How about Macrium Reflect? Just three utils that make the PC experience very slick, and it would be pointless running in a VMunlessthat VM is the main desktop. Which kind of defeats the point.

author: nctnico
author_rank: Super Contributor
timestamp: October 17, 2022, 12:36:20 pm
content: Quote from: dunkemhigh on October 17, 2022, 10:49:07 amThose utils you mention are actually very old concepts that are available on Linux (Unix) as standard for a long time.

author: brucehoult
author_rank: Super Contributor
timestamp: October 17, 2022, 12:46:29 pm
content: Quote from: nctnico on October 17, 2022, 12:36:20 pmI've never heard of any of them but it seems the first is just some kind of Spotlight clone, the second is roughly an enhanced Midnight Commander, and the 3rd backup.So, nothing special?

author: PlainName
author_rank: Super Contributor
timestamp: October 17, 2022, 01:37:34 pm
content: QuoteKind of. Its main benefit is to integrate into the standard OS file dialogs to quickly change folder. For instance, the app alway sets file open to ~/appdata/app/user or similar, whereas I've just saved (from another app) to r:/temp. One click change. Of course, there are favourites, searching and all that stuff, but they are not the primary use.It's greatest feature, that I use alot, is to use some other file explorer app/dialog. For instance, you choose file save and the app opens up the last folder it saved to (say, ~/long path/project1). But you in DOpus you have open ~/long other path/project20. No problem, Listary will instantly change the file save dialog to the project20 path. You don't realise how useful this is until you use it (kind of like mouse wheels and back buttons).QuoteYou could say Midnight Commander is roughly an enhanced Explorer. There is much more to it than that. Much more.QuoteWhich shows you don't realise what you're missing. Yes, it's a backup (and restore - don't forget that). But it write images. And significantly it can do that on a live system - this uses Windows shadow service, but there are similar that use their own variation of same. There are none that can do that on Linux - if you have open files you can't back them up.Further, it does incremental and differential images. Significantly, it knows which main a diff or inc comes from. That is, you can do a main on media1 and a main on media2, then diff on media1, another diff on media1, then a diff on media2. The diff on media2 will be correct and completely unaffected by whatever is on media1. And a further diff on media1 will be unaffected by whatever is on media2. Ditto for incs, of course.Result is that you can have multiple coherent backups, utilising differential and incremental stategies, completely independent of, and unaffected by, each other. And do that without having to pop off for a coffee while it happens.

author: PlainName
author_rank: Super Contributor
timestamp: October 17, 2022, 01:49:33 pm
content: Forgot to also say that even if Linux had those exact tools available, they would be useless because that kind of thing doesn't cross the OS-VM interface. Which is, really, the point of a VM. And a reason I dislike having to run some often-used application in a VM, because it is difficult to use all these background productivity utilities that people seem not to notice exist.

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 17, 2022, 01:56:45 pm
content: Quote from: dunkemhigh on October 17, 2022, 01:37:34 pmEh? Why would having a file open hinder backups at all?Also, if you want snapshots, just use LVM2. Atomic instant snapshots of the filesystem state at any given point in time, without blocking or pausing access to it, like on a parallel timeline; no filesystem support for it required either. Doesn't even matter if the filesystem is being used by a virtual machine. Do your backuppy stuff, even incremental or tar-based, and then drop the snapshot. Trivial. Well over decade-old, stable stuff.I think you've just found a set of tools you like. That's fine, nothing wrong with that. Just don't go spreading lies about what "Linux cannot do", please.

author: tggzzz
author_rank: Super Contributor
timestamp: October 17, 2022, 02:17:02 pm
content: Quote from: Nominal Animal on October 17, 2022, 01:56:45 pmJust so.And preferably do it on another forum better suited to the usual flame wars

author: PlainName
author_rank: Super Contributor
timestamp: October 17, 2022, 03:04:33 pm
content: QuotePrevents access to the file, so it's missing from the backup.QuoteIs there a backup util that replicates the Reflect features, whatever the underlying details? Every time I ask there appears not to be.QuoteI would hardly use every day a set of tools I disliked! (Hence why I am not on W10, of course.)But you could also look at it as discovering the electric kettle when everyone raves about making a cuppa on a camp fire. That would be 'just a tool I like', wouldn't it?QuoteBelieve me, I would be deliriously happy if someone would come along and show Linux doing those things, because then I would really consider jumping ship. The backup is the major road hump at the moment (well, since forever really), and if there were a simple util that replicated Reflect that would go a looooong way. There is a company (Terrabyte) that makes a similar tool for Windows and Linux, but the Linux version cannot make an image while the OS is running. Since they surely know what they are doing (the Windows version does it, and they used to even manage it without the Windows shadow copy thing) I have to assume that there is some reason why it isn't possible and, as I keep pointing out, no-one has shown me that it is possible.

author: PlainName
author_rank: Super Contributor
timestamp: October 17, 2022, 03:07:27 pm
content: QuoteI am not trying to start a flame war. In fact, I would love it if I was shown to be wrong. But, OTOH, you don't seem to have a problem with someone propagating the myth that a VM is the answer whenever sticking with Windows comes up.

author: tggzzz
author_rank: Super Contributor
timestamp: October 17, 2022, 04:39:57 pm
content: Quote from: dunkemhigh on October 17, 2022, 03:07:27 pmDo you make a habit of deliberately snipping the relevant context thereby misrepresenting other people in order to make a point? That's called a "strawman argument", and is generally regarded as being "suboptimal behaviour".Here's the full context,https://www.eevblog.com/forum/microcontrollers/partcl-command-languaje-for-tiny-microcontrollers/msg4468279/#msg4468279and everybody can see that I wasn't even replying to you!EDIT:  your posting style coupled with the slightly provocative moniker tickled my memory. It seems you do have form in that respect, e.g.https://www.eevblog.com/forum/supporters-lounge/legalities/msg4415617/#msg4415617https://www.eevblog.com/forum/chat/important-poll-should-there-be-an-off-topic-section/msg4410943/#msg4410943

author: woofy
author_rank: Frequent Contributor
timestamp: October 17, 2022, 05:26:59 pm
content: Quote from: dunkemhigh on October 17, 2022, 01:37:34 pmAs far as I know, Linux copies open files just fine. In fact I'm not sure there is a way to stop it doing so.

author: PlainName
author_rank: Super Contributor
timestamp: October 17, 2022, 07:07:32 pm
content: QuoteI quote what I am commenting on as succinctly as seems relevant. If you want to see more it's easy to find, but entire threads of quoted quotes of quotes is pretty bad.If you think the context is missing then it's either because you don't appreciate the point of my argument, or because I missed some context that you consider important. There is no deliberate misrepresentation intended, or implied.

author: PlainName
author_rank: Super Contributor
timestamp: October 17, 2022, 07:09:53 pm
content: Quote from: woofy on October 17, 2022, 05:26:59 pmCan't they be read locked?Perhaps that's not the cause, then. It may be that 'image' is the show-stopper.

author: tggzzz
author_rank: Super Contributor
timestamp: October 17, 2022, 07:53:58 pm
content: Quote from: dunkemhigh on October 17, 2022, 07:07:32 pmYou should be as succinct as possible, but not "succincter".Whether or not you intend to misrepresent people is irrelevant.You do misrepresent them. Repeatedly. Deliberately. I demonstrated that with the links I gave and which, surprise, surprise, you snipped.You make it so difficult for all the readers to find that they won't bother - to your benefit. Thatattitude is difficult to distinguish from that of a troll, which arguably isn't inconsistent with your moniker. OTOH if there is a reason for.your moniker, I'm curious to hear it!Make it easier for your readers and yourself by doing less, i.e. by not snipping context. Go.on; I dare you!

author: PlainName
author_rank: Super Contributor
timestamp: October 17, 2022, 08:36:33 pm
content: QuoteBased on a different moniker (which doesn't lend itself at all to conspiracy theories), it was the one that managed to skip all the 'sorry, that already exists' filters whilst still being loosely related and all I could come up with on the spur of the moment. It is entirely meaningless except for being obscure unless you know my other IDs.

author: woofy
author_rank: Frequent Contributor
timestamp: October 17, 2022, 08:43:50 pm
content: Quote from: dunkemhigh on October 17, 2022, 07:09:53 pmI don't know of a way to read lock a file.Files can be protected against being written/modified/deleted (even with admin rights) by setting the i flag, but they are still readable.

author: tggzzz
author_rank: Super Contributor
timestamp: October 17, 2022, 08:47:24 pm
content: Quote from: dunkemhigh on October 17, 2022, 08:36:33 pmFair enoughMake it easier for your readers and yourself by doing less, i.e. by not snipping context. Go.on; I dare you! Just use the 'quote' button at the top right of the relevant message, which automatically inserts a link to the message, then add your contribution.

author: PlainName
author_rank: Super Contributor
timestamp: October 17, 2022, 08:52:41 pm
content: Quote from: tggzzz on October 17, 2022, 08:47:24 pmSo how do you know which bit I am commenting to? Is it my username, your suggestion, your acceptance, something else?

author: tggzzz
author_rank: Super Contributor
timestamp: October 17, 2022, 09:21:15 pm
content: Quote from: dunkemhigh on October 17, 2022, 08:52:41 pmI was taking your statement at face value.Your subsequent response indicates that youarea troll, as per my suspicions I voiced inhttps://www.eevblog.com/forum/microcontrollers/partcl-command-languaje-for-tiny-microcontrollers/msg4468492/#msg4468492

author: PlainName
author_rank: Super Contributor
timestamp: October 17, 2022, 11:36:09 pm
content: I think you're being the troll hereI tried to demonstrate the problem with your suggestion (and why I use short quotes) and expected either a recognition of that or a solution to it. Yet pointing it out and asking how it's meant to be resolved gets me labelled a troll? Perhaps you'd best put me on your ignore list, then, and we can both be happy.

author: MK14
author_rank: Super Contributor
timestamp: October 17, 2022, 11:44:14 pm
content: I don't think anyone, who has already posted in this thread, is a troll or trolling at all. There just has been perhaps, a bit of miscommunications, and misunderstandings, to a limited extent.

author: slugrustle
author_rank: Frequent Contributor
timestamp: October 18, 2022, 06:00:17 am
content: I'm working on something quasi related to this. It's a board with a microcontroller (PIC18F45Q43) that runs 16 ordinary 3.3V digital outputs and 6 PWM outputs—these share the same common / negative reference that is isolated from earth (floating). There's also 3 button inputs, a trigger input, a trigger output, and an RS232 link to a computer—all of these things are earth referred.The computer can send instructions to the MCU over the serial link. The MCU stores these in RAM and executes them after reception. However, the idea is to load a whole "program" or sequence of instructions in one go and execute that until completion, then wait for a new program or for the user to hit the repeat button. The cycle time for instruction execution, especially instructions not involving the serial link such as modifying the digital outputs or PWM periods or duty cycles, is very quick. My main use case is to test circuit boards for which full firmware has not been developed yet in my day job as a hardware-focused embedded engineer.I love assembly language, so the command language looks a lot like assembly, and I'm writing the firmware for this in assembly. I chose an 8-bit PIC because I think they're fun to program in assembly. The attached firmware is perhaps 70% done. First pass implementation is in, and some simple instructions work, but I have a lot more testing to do. I also want to put more self checks in the code, like RAM test and main clock frequency test.Apropos of nothing, I was really disappointed when Microchip deprecated MPASM, but I'm getting used to PIC-AS. The github repohttps://github.com/dsoze1138/MPLABXv5xx_pic-as_exampleswas very helpful, as was Microchip's various documentation on PIC-AS, including migration guides.

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 18, 2022, 07:27:49 am
content: Quote from: dunkemhigh on October 17, 2022, 03:04:33 pmNo, it does not. Nothing prevents you from reading a file, even if it is being executed/mapped/accessed by another process. Whatever gave you that idea?I do know thatWindowsprevents one from deleting or renaming files being executed, but Linux has no such limitations. The one limitation is you cannot open an executable for writing if it is being executed right now; but you can open it for reading, and delete and rename it just fine.Quote from: dunkemhigh on October 17, 2022, 03:04:33 pmI don't know, because I do not know what features Reflect provides.Besides, I've never used "backup utils" in Linux. It is so simple stuff (to me), that I just set up whatever is convenient on each machine for me. I spend more time thinking about what kind of backup policy is needed and works best, than implementing it.(On one server, running a black-box Java application, reverse-cached to open internet using Apache, I used a script that polled the number of active TCP connections to the server during the backup time window each night, moving the snapshot forwards every time there were fewer users; and then just tarred the entire dataset off the snapshot to a backup machine for a dailystablesnapshot. I had a pair of identical servers in a rack, also connected directly to each other using fixed IP addresses.)The fact that you're looking for a "backup util", also means that you're looking for someone else to worry about the policy details for you. It's not how Linux works, is all.Quote from: dunkemhigh on October 17, 2022, 03:04:33 pmI don't know, because I don't quite get the analogy there. (I'm being serious, not snarky here. If you detect any kind of snarkiness in my tone in this or my previous message, please let me know and I'll try to reword to delete such unwanted undertones.)The way I see it, is that Linux is like a raw, modular kit of tools, whereas Windows and macOS are polished toolcarts. There is nothing wrong in preferring to use a good toolcart;they work. But judging Linux based on what kind of toolcart modules (readymade utilities) it has available, is like rating a car based on how well it drives on two wheels. That's not what it is for.Quote from: dunkemhigh on October 17, 2022, 03:04:33 pmThere being a ready-made utility for it is completely different to it being possible. You're assuming that the lack of an utility means the lack of capability. This is the core untruth here.I recommend you take a look at theKernelTalks How-to guide: LVM snapshot.To set up LVM, instead of accessing storage devices and their partitions directly, we assign them to one or more LVMVolume Groups. (A VG can span devices, which can be very useful if you're running out of space on one.)In each Volume Group, we can create Logical Volumes. You create one LV for each filesystem you want to use. (I typically use one for /boot, one for operating system, one for /home (user home directories), and on a dedicated server, one for the service data (/data or /var/data or wherever is applicable). Each LV looks very much like a storage device partition to the user, and are formatted and used exactly like partitions are.The key thing here is to not use all of the space on the Volume Group for the Logical Volumes. Whenever we create a snapshot of a LV (and thus the filesystem in it), unused space in the Volume Group is used to store the divergence between the snapshot and the Logical Volume it was taken from. (In thick provisioning, we use the unallocated parts of the Volume Group; in thin provisioning, the unused blocks in the Logical Volumes on that Volume Group too –– very much like memory overcommit.) We can always add more space to the Volume Group by adding another storage device to it, though.This is the first design issue that one must carefully consider when installing a Linux distribution on their machines, if they intend to use LVM snapshots. What is the provisioning policy to be used? No easy answers, no rules of thumb really (except "20% is a good number"). This is the stuff the most experienced Linux sysadmins talk shop about when relaxing and having favourite beverages with colleagues.It becomes even more complicated when you do odd stuff like Raid-0 and Raid-1 on different storage devices for added throughput or redundancy. (Yes, I do like the idea of two or four Samsung 980 Pro NVMe drives.) In such cases, using a filesystem that itself provides snapshot capability (btrfs, zfs) can make things simpler –– but even then the backup machinery itself stays logically the same, you just use slightly different commands, and provision for the snapshots in a slightly different manner.Okay, let's say you have done all this. You have say/homeon an LVM Logical Volume, in a Volume Group with sufficient free/unused space.To back it up, you create an LVM snapshot, usinglvcreate -Lspace-slogical-volume-device-nsnapshot-name.Thespaceis the amount of space needed for the differences between the snapshot, and the original Logical Volume. This, too, is a policy issue, and heavily depends on how much data you expect could be changed on the original Logical Volume during the lifetime of this snapshot.The original filesystem on that Logical Volume is not impacted at all by this: it does not need to be unmounted, unused, or even idle. It is important that it is in more or less quiescent state for the filesystem snapshot to be useful for us humans –– we want files in their completed state, not in the middle of being modified ––, but it is not a technical requirement at all, just a human one.The snapshot itself becomes a new Logical Volume. You can treat it as raw partition data (for filesystem imaging), or you can mount it for full, selective, or incremental backupping. You can even mount it read-write!I myself try to take the snapshot when there are as few applications or users accessing the filesystem as possible. This is to ensure that the snapshot I get is useful to us humans, instead of containing e.g. half-edited versions of files. There is no technical requirement to do so, though. I then mount the snapshot somewhere useful (often I create a temporary directory as a mount point at/mnt/snapshot/snapshot-nameor/tmp/uacctname-snapshots/snapshot-name), take my sweet time in tarballing the interesting contents (usingniceandioniceto make sure it does not slow down other processes, since there is no hurry here), and finally unmount and remove the snapshot (umountmountpointfollowed bylvremovevolume-group-device/snapshot-name).That's all there is to it. The key points are the design choices: using LVM on the storage device, assigning it to a Volume Group; which mount points to make their own filesystems and LVM Logical Volumes; how much space to leave unassigned for in the Volume Group for use by snapshots; and how much space to reserve in each snapshot for the difference between the snapshot and the filesystem it is a snapshot of.After all that is decided, the backup itself can be a simple dozen-line Bash shell script you trigger automagically by cron/anachron/at. Or you can make it more intelligent, for example keeping a remote storage in sync with the latest snapshot using rsync, or whatever you need.There are so many options and choices here that I really don't even see how a single utility forcing a specific model on me would be useful. Sure, it'd be useful in the same sense that one of those capsule coffee makers is useful: just drop in a capsule, plug it in, add water, and turn it on to get a cup of coffee. But Linux isn't like that: it isn't a cohesive tool set, but a modular set for you to shape. (It is up to us humans to decide whether that's good or bad: it definitely doesn't work for all. I personally even have a Dolce Gusto capsule coffee maker, but a two-pot Cloer 5548 tea maker!)Besides, everything really starts at the point you initially install the Linux distribution anyways, because that is when you make the critical design choices. Sure, technically, one can do them afterwards too, but in my experience, it is just faster to save your files and customizations, and reinstall the entire system, and then copy your files and customizations back. I do this about once a year or two anyway, because I choose even my Linux distro based on my workflow at that point in time.I really do not want this to sound snarky, but just a matter-of-fact casual affirmation: If one is not interested in making such design choices and implementing such stuff themselves, then Linux really isn't well suited for them anyway.Linux is not, and never will be, simply an alternative to Windows or macOS. Everything that makes Linux powerful for types like myself is the fact that it does not do those design choices for you –– although some software, like gnome and systemd, do insist on their developers being right in doing so. Like so many things in life, that very key facet is a strong point for some, and a fatal weak point for others.Apologies for the long-ass wall of text.

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 18, 2022, 07:37:04 am
content: Quote from: slugrustle on October 18, 2022, 06:00:17 amJust out of interest: are the instructions text, or binary? (Edit: looks like single-byte binary instructions, based on the attached files.)A bytecode interpreter can be much simpler than a text parser, and obviously more efficient, both space and time wise.On the computer side, the user interface can incorporate the bytecode compiler/decompiler, so that the end user is always dealing with just plain text code, but it does increase the overall complexity a bit. (In particular, personally I'd avoid the decompiler part, meaning that code already sent to the device cannot be read back in human-readable form.)Your use case sounds quite similar to mine, except that the MCUs I use all have native USB, and many of my use cases revolve around USB connectivity (like simulating USB devices I don't have by reusing USB data captures). I have wondered whether switching to a bytecode (with the compiler integrated into an easy-to-use UI on the computer) would be more useful, thus this question.

author: tggzzz
author_rank: Super Contributor
timestamp: October 18, 2022, 08:44:32 am
content: Quote from: MK14 on October 17, 2022, 11:44:14 pmI am very reluctant to think of anybody as a troll.Nonetheless, thespecific natureof the miscommunications, in this threadand on other threadsleads me in that direction.

author: nctnico
author_rank: Super Contributor
timestamp: October 18, 2022, 09:11:16 am
content: Quote from: tggzzz on October 16, 2022, 10:13:31 pmWhy would you need to tidy up printf over and over? The whole command line, mini C-library, etc I use is like a blob I can throw onto a(ny) microcontroller. There is no re-doing.Quote from: Nominal Animal on October 18, 2022, 07:37:04 amThat starts to sound awfully close to using remote debugging by feeding the microcontroller instruction-by-instruction. Realistically: how many high-level commands would you really need to test something? On average I have 3 commands to test a software component in a microcontroller.

author: brucehoult
author_rank: Super Contributor
timestamp: October 18, 2022, 09:17:47 am
content: Quote from: Nominal Animal on October 18, 2022, 07:27:49 amIn particular, note that while you can't write to an executable file that is being executed, you can *replace* it with a new version of the program, with no outage window at all (not even microseconds) by creating a new executable file and then using the rename(2) system call to delete and replace it in one atomic step. (Not actually delete if there is another hard link to it somewhere, of course)

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 18, 2022, 11:01:46 am
content: Quote from: nctnico on October 18, 2022, 09:11:16 amThis is not about testing the microcontroller itself, but about using it as a tool for investigating something else.Even when it is about the microcontroller itself, it tends to be about finding useful (default?) settings for controlling external devices, from sensors to actuators. Things like maximum pulse frequency, clock rates, and such.

author: slugrustle
author_rank: Frequent Contributor
timestamp: October 18, 2022, 05:13:30 pm
content: Quote from: Nominal Animal on October 18, 2022, 07:37:04 amThe instructions are binary. Particularly, each "opcode" is a single byte, but it can have no argument, a single byte argument, multiple multi byte arguments, etc.I send the instructions over serial as ASCII hexadecimal, but you could send raw bytes. They're stored in linear byte-addressable RAM, and there is some minor post-processing to check validity and adjust some things for easier processing:Check that loop targets are within the program's address rangeI want to check whether loop targets are instruction "opcode" bytes and not argument bytes, but that is more involved... maybe later.Check that digital output values are set before digital outputs are enabled.Check that PWM period and duty cycle are specified before a PWM output is enabled.Check that the final instruction either ends the program (and waits for a new one) or loops the whole program.Modify loop offsets and iteration counts. I specified the instructions so that I can write them by hand; these modifications make processing faster on the MCU.After post processing, if the program is valid, it's interpreted in a big loop that has a jump table based on the "opcode" byte for each instruction. The jump table goes to a section of code that does what is required for the instruction, updates the indirect addressing registers so they point to the next instruction's "opcode", and branches back to the interpreter loop.This is just the first thing I thought of given my familiarity with PIC microcontrollers, love of assembly, and use case. Your comment is the first time it dawned on me that this thing is a bytecode interpreter.

author: slugrustle
author_rank: Frequent Contributor
timestamp: October 18, 2022, 05:38:12 pm
content: Quote from: Nominal Animal on October 18, 2022, 11:01:46 amNominal Animal is on point, and maybe I wasn't clear. At work, I'll get these circuit boards that have all sorts of hardware circuits I or someone else designed that interface with an MCU, and I have to test at least some of the hardware circuits—the firmware is developed by a separate team. The MCU will either not have firmware, or the firmware will run the thing per design intent. When I'm testing, I often need specific sequencing of outputs to test the hardware circuits, so this thing replaces the MCU on the board to drive the hardware (well, not replaces, I just wire the MCU on the board in reset).Previously, I had a little board like this with a few digital outputs and would modify the firmware for each test. That works fine, but I think this will be a bit faster and more productive, since I have a limited set of things that I typically need to do. Basically, it's a glorified digital pattern generator. I spent a long time looking for a digital pattern generator or digital I/O card or digital I/O box that would be easy to program for my tasks, but products on the market often have few to no PWM outputs, insufficient PWM resolution, limited I/O drive strength, unspecified latency, a cumbersome API (or heaven forbid, a graphical interface), or something else that I wasn't fond of.It's nice when you can make your own tools. After I get this board up and running and have some time with it to figure out what I'd like to change, I plan to make a version of it where each digital output is isolated, and the output driver circuitry lives on a little circuit board connected to the main unit with an RJ50 cable. The idea would be to supply power to the output driver board, use RS485 or LVDS for the faster digital signal, and use single ended logic for slower stuff like output enable/disable. That would allow me to swap the driver board for each output to choose 5V output, 3.3V output, open drain output, back-to-back MOSFET output, gate drive output, relay output, etc. as needed. The isolation would remove any concerns about ground shift and even let me simultaneously test circuits referred to different power domains. Right now, it's a pipe dream.

author: slugrustle
author_rank: Frequent Contributor
timestamp: October 18, 2022, 06:40:56 pm
content: Quote from: Nominal Animal on October 18, 2022, 07:37:04 amHere's the cpp file I used to send the first program to my board over serial for testing. It's a bit of a mess, and the program is hardcoded, but it let me run the test! Eventually, I hope to write a little command line assembler for the byte code language. Only real differences from the actual bytecode would be mnemonics, labels for loop targets, and ability to use different radices for arguments.Although it might be nice to learn pdcurses or similar and allow some interactivity with the program once it started. The bytecode has some instructions that wait for reception of a particular byte from the host computer.

author: PlainName
author_rank: Super Contributor
timestamp: October 18, 2022, 09:02:26 pm
content: Thanks for taking the time and putting in the effort to make a comprehensive post.Quote from: Nominal Animal on October 18, 2022, 07:27:49 amOK, I am happy to be told I'm wrong about that. As I said, something prevents the live backups and that's the only thing I could think of that might. File access can be prevented - those flags doe something! - but I accept they aren't the problem here. Dunno what might be, but clearly something else is.QuoteBriefly:Creation of imagesdiff/inc of specific existing image (and associated incs)mounting of image (and inc/diffs) for file & folder accessQuoteIs there a problem with that? If that's not how Linux works then Linux is unsuitable and that, there, is surely a good enough reason not to use it. However, let's assume that we are beyond the 'real users use the CLI' thing and accept that the GUI is actually useful. Simillarly, applications are actually useful, and users that want to get something done instead of spending their time making the tools to do something (if they ever get around to it) are perfectly OK users.Frankly, I don't want to mess around at a low level making mistakes. For something like a backup I DO want someone else - that knows what they are doing - to supply something that works. Making sure diffs are a) created properly and b) applied to a backup to create a later restore are not trivial operations, and I pay someone so that I can use their knowledge, experience and skillset instead of finding I can't restore some file. I don't see a problem with that. If it makes me a Windows user and not a Linux user then that just says Linux is not suitable and explain the cause of my problemQuoteThe implication from your post was that I happen to like what I've been using (which is a perfectly accurate observation). Typically, this is said about someone who knows only a hammer so treats everything as a nail to be hit. My reply is that an alternative view might be someone discovering the electric kettle and liking it, while everyone else is still using fire. Isn't the kettle a better thing for making tea? And yet, it might appear that I only like the kettle because that's what I use, missing the fact that it really is a better tool.QuoteI think that's a bit, er, not pedantic... perhaps nitpicking. If there is no tool to 'do it' then it is effectively not possible (for me, for reasons stated above).QuoteThanks for that info. I can see it would replicate the Windows shadow service to some extent.QuoteThat gives you access to a snapshot of the file system, right? But can you create an image from that? I don't think you can, but I am not au fait with this stuff (as you may have noticed).Also, you have a snapshot but how do you create a diff with a previous backup? Or an inc with previous backup and associated incs? This is where I really, really appreciate someone else taking care of such detail.QuoteWell, that's fine for your purposes but, as noted, that approach doesn't work for me. I am still not sure you replicate what I am after, only that you can create a snapshot and copy it off somewhere.QuoteThat leads me to ask: do you write your own compilers? Editors? Browsers? I think it would be safe to say you let someone else create those apps and utils. They may just tie together modular parts, but isn't that what apps do?QuoteThis is very true, and without trying to start a war, I thought that was not the Linux way. When I was seriously looking at moving over to Linux (before this backup business killed it) I didn't know exactly which distro or setup I would want, so I figured I would just have a kind of dual-boot thing where each distro would access the same home folder and similar stuff. But, no, that doesn't work - set up home for one distro and another distro can't handle it.It seems to me that if I knew how to drive LVM to achieve my aims, I would have tostartwith the proper setup. Should that turn out to be inappropriate, I would have to go back and start again. I think that would be really painful, particularly if I got a fair way in before realising I'd made a mistake.QuoteNot a problem. It was a useful read, and if nothing else it kept me away from the forum while I perused it

author: brucehoult
author_rank: Super Contributor
timestamp: October 18, 2022, 10:37:51 pm
content: You might want to look into rdiff-backup. I've been using it for about 15 years.>But can you create an image from that?You can of course create a disk image from any directory tree, anywhere.

author: nctnico
author_rank: Super Contributor
timestamp: October 18, 2022, 11:22:29 pm
content: Actually , for Linux a disk image is not really necessary. An archive with all the system files is sufficient to restore a system.

author: tggzzz
author_rank: Super Contributor
timestamp: October 18, 2022, 11:49:18 pm
content: Quote from: dunkemhigh on October 18, 2022, 09:02:26 pmThat doesn't match what you said earlier, where you presumed that a difficult issue in Windows (backing up an open file) was also a difficult issue in Linux. It took several people several times before you accepted you didn't understand how it wasn't even an issue with Linux systems.The analogy would be of someone who used screws and knew nails existed, didn't understand the significant differences, and was convinced that a problem with screws was also a problem with nails.As a separate.point, your presumption that someone that creates a pretty GUI knows how to implement the core functionality correctly is, um, touching.The Linux mentality is to create a tool that does one job well, and then allow other tools (e.g. a sugar-coated GUI) to use it as part of a complete customised applicatkon. Windows has tried to emulate that extendability and composability with PowerShell but has, I am lead to believe, only partially succeeded.

author: PlainName
author_rank: Super Contributor
timestamp: October 18, 2022, 11:59:32 pm
content: Quote from: nctnico on October 18, 2022, 11:22:29 pmFor that matter, it's only really useful to capture the Windows system drive (mainly for the registry). And, if it's later than W7, the non-drive partitions that the OS finds essential and which most users don't know exist.But the big advantage of an image, over file and folders, is speed. And, when it really hits the fan, not needing an OS to do a restore (although I don't know how relevant that is to Linux, particularly nowadays, other than to note that the backup utils I did take a look at needed to be running in order to work).

author: PlainName
author_rank: Super Contributor
timestamp: October 19, 2022, 12:07:22 am
content: Quote from: tggzzz on October 18, 2022, 11:49:18 pmThat's misinterpreting my post. Perhaps even taking out of context.Earlier, I said the problem with taking an image on Linux is that none of the things I've tried, or looked at, can take an image of a live system. I gave the specific example of Terrabyte who do the same backup app for Windows and Linux, and whereas on Windows it will image a live system on Linux it won't. I also said I didn't know the details of why, but pointed out that if it could be done then I'm sure Terrabyte would do it (because it's a big selling point). I offered a guess that it might be file locking, and accepted it isn't when told that wouldn't be an issue on Linux.(And, you might notice that Linux can prevent access to files, so the supposition that it might be a problem isn't completely stupid.)Do I need to point out, again, that I don't use Linux so I'm hardly a Linux guru. One is allowed to hazard a guess, isn't one?

author: tggzzz
author_rank: Super Contributor
timestamp: October 19, 2022, 12:15:21 am
content: Quote from: dunkemhigh on October 18, 2022, 11:59:32 pmAh yes, the windows registry. What a mess; all applications and all users config scrunched together into one unholy hairball. In Linux all my config is contained in the same place as my files, i.e. my /home/make directory. The only thing I bother to backup is my home directory, ditto copying it to a new machine. All the other config doesn't need to be retained, either because it is professor specific or specific to the os/application version.As for having a running operating system before using any restore utilities, that's trivial. Just use any installation live disk running from USB drive.and RAM, or a specialised recovery/rescue disk. I've.used those to rescue Windows when Windows couldn't rescue itself.

author: tggzzz
author_rank: Super Contributor
timestamp: October 19, 2022, 12:23:22 am
content: Quote from: dunkemhigh on October 19, 2022, 12:07:22 amYou made explicit statements about (imagined) problems with Linux; you didn't merely hazard a guess.The concept of using the same application (Terrabyte) on wildly different systems to something as system specific as backup and restore is, to put it mildly, misguided. Either you end up with the lowest common denominator functionality, or you end up with something that works.well on one system and poorly on the other. Doubly so when considering the many different Linux filesystem each with their own unique advantages.

author: brucehoult
author_rank: Super Contributor
timestamp: October 19, 2022, 01:38:52 am
content: Quote from: dunkemhigh on October 19, 2022, 12:07:22 amOf course you can image a live Linux system!The problem, if any, is continuing to do work on the system WHILE it is being imaged, in which case related files that should be in sync with each other can be backed up at different times, with one copied before some transaction and another copied after it.That is a completely OS-independent problem.On 'doze I believe it will only be able to backup a live and actively used system correctly if using Volume Shadow Copy Service introduced for the NTFS file system in XP/Server 2003, not for example on any kind of FAT variant.Similar functionality is available in *nix filesystems too, such as btrfs, ZFS, Fossil. I believe LVM technically adds block-level snapshot ability on top of other filesystems such as ext4, but you're usually better off using one of the previously mentioned filesystems if you want snapshotting.And also, of course, Apple APFS (introduced in 2016 in OS X 10.12 "Sierra") supports snapshots.If Terabyte doesn't support backups of live Linux systems using [btrfs, ZFS, Fossil] file system then it's only because they are lazy or incompetent or no one on Linux finds enough value in their product to buy it.

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 19, 2022, 03:08:56 am
content: Quote from: dunkemhigh on October 18, 2022, 09:02:26 pmNo, I'm just saying that you will be more efficient with something else than Linux. I'm really referring to theUnix philosophyhere, and how backups in Linux can be constructed in so many different ways, that asingleutility could not cater to more than a small fragment of users.Quote from: dunkemhigh on October 18, 2022, 09:02:26 pmI don't see a problem with that either. The problem is that the someones capable of doing that, like myself, are experienced server sysadmins, and not making $40 utilities. If I were to do what you asked, the result would be a guide, not an utility. The mismatch (of you wanting something that is not offered by the market) is just a result of how this particular market works.Quote from: dunkemhigh on October 18, 2022, 09:02:26 pmI knew it! I should have written "I think you've found a set of tools you are comfortable and effective with", because that was what I was trying to say AND imply. No hammer analogy association was intended; I even used "a set of tools" for that reason (and because you are not trying to use a single tool for everything, as you described different ones for different tasks).As to the analogy itself: you see, I'm a "traditional" Finn, and I actually love making coffee over a camp fire: it's traditional, especially in the winter, during ice-fishing trips. Somehow, it just tastes better than when made with a Moccamaster at home, or brought along in a thermos flask. (The reason is, of course, the same why food tastes better when you're hungry, and just about nothing to do with the tools used. Emotional context, fullness of experience, romanticism, etc.) I honestly didn't intuitively grasp the analogy you were going for, because of difference in personal experiences. (Before you ask, during the summer, bringing a pan and a milk carton full of crépe batter, is a must also.)Quote from: dunkemhigh on October 18, 2022, 09:02:26 pmNo, because we are not talking about something that is a single action, coverable by a simple utility, in Linux. There are many, many options, and a single tool just does not cover them all. (But I do have to admit that even if such a tool did exist, I probably would not know about it.)And "effectively not possible for dunkemhigh" is definitely not the same thing as "cannot be done in Linux", which is what I objected to. Do you really disagree with that objection?I tried to explain this by showing how some of the details are design choices really made at OS installation time, when LVM is used.And LVM snapshots are just one of the many approaches one can take. For example, if I wanted to monitor the directory tree I keep my email, and another where I keep my projects, I could easily write an utility that monitors them for changes, uses file leases to check when they are not being accessed (and delay any access attempts up to a few seconds if necessary, so that a snapshot can be taken on a per-file basis), and makes a synchronous incremental backup to a remote device.(If you want, I can write another long post about how I once found a way to track changes made to system settings and blame them on specific humans, even when they all preferred to act as root/superuser (sudo su -), without anything visible in the file system itself. I got annoyed at people making stupid changes and then going to a long weekend, without telling anyone about what they did, you see. Nowadays there are two completely different subsystems in Linux that allows that: one is the audit subsystem, and the other is the fanotify subsystem. And then there are process-specific stuff like LD_PRELOAD library interposing and even the seccomp syscall filtering facilities that could be used instead, too... Lots and lots of options and choices.)Quote from: dunkemhigh on October 18, 2022, 09:02:26 pmThe newly created device (/dev/volume-group/logical-volume-name)isthe binary image of the filesystem.Depending on exactly what filesystem you use on it (ext4, xfs, btrfs, zfs, etc.), there are filesystem-specific tools that can compress the image (so that you don't store unused blocks), or convert to some other form used by other tools.Quote from: dunkemhigh on October 18, 2022, 09:02:26 pmThere are filesystem-specific tools for this, but on the mounted filesystem, you could userdiff, for example.One approach I've used is to have a listing of all the files in the backup, including the sizes, modify timestamps, original inode numbers, and a SHA256 hash of their contents. The script or program doing the backup checks all files in the snapshot, and if the size, modify timestamp, or inode number differs, then the file is hashed and checked if it has been modified. If it has, then it is copied to the backup and the listing updated. Note that this is then incremental per filesystem, but complete regarding files. (One could usediffandpatchfor incremental changes to individual files.)Again, there are many utilities and tools you can do this with.Quote from: dunkemhigh on October 18, 2022, 09:02:26 pmTrue. Note that I never even implied that Linux can give you a workflow that you like, only that all of this is possible (and actually common on the server end; that's where I learned about all this stuff, after all).Quote from: dunkemhigh on October 18, 2022, 09:02:26 pmWell, I consider those individual tools.The difference is more like between "I use Visual Studio" and "I'm on x86-64, using make 4.2.1, gcc 9.4.0, binutils 2.34, and glibc headers from libc6-dev 2.31".What you see as a trivial task – just back up my files, will ya? – is a huge set of alternate approaches to me; like the difference between an integrated development environment, and a set of toolchain tools.Quote from: dunkemhigh on October 18, 2022, 09:02:26 pmTrue. This would be very painful to learn to do on an installation one wants to use for everyday work.None of what I've written above should be considered something I suggest you do, only as a proof that it is indeed possible (because some of us Linux "long-aired smellies" do do it).It would be very stupid of me to not give anypracticalsuggestions to those in dunkemhigh's shoes. As I use my own stuff, I cannotrecommendany specific readymade utilities, but I definitely can give you links to existing utilities that others do use, and give my own comment on them. Although I provide links, do note that you want to use the package manager in your preferred Linux distribution to install it (and not all may be available in all distributions), and not download something willy-nilly off the internet. In no particular order:rsync– the incremental file transfer toolThis lets you create a remote mirror of the contents of some part of your filesystem, or synchronize two filesystems only transferring the changes between them. The tool itself is a command-line one, but there are several graphical user interfaces:grsync,BackInTime,Kup,LuckyBackup.Déjà Dup– simple backup tool for GNOMEA graphical user interface forduplicity.MintBackup– a simple backup tool for Linux MintVery simple backup tool for Linux Mint users and their personal home directories. Can also record the installed packages, making reinstallation to the same configuration easier.TimeshiftandBackInTimeTimeshift is for system only, but especially interesting if you use btrfs. BackInTime is what Timeshift developers recommend for user files; backintime-qt is the graphical user interface for it.Restic– command-line, but cross-platform (Linux, Windows, Mac, *BSDs)Seedocumentation. Documentation has an example of how to set up backups to Amazon S3.AmandaandBacula– client-server solutionsThese are the solutions larger organizations start looking at for backup solutions, and consists of a server-side component (used where the backups are stored), and a client-side component (used on backed up machines).

author: sarbog
author_rank: Contributor
timestamp: October 19, 2022, 03:17:00 am
content: There is no perfect language. I prefer C since it is close to the hardware for embedded stuff. I debug using STM32 Cube Ide .. not perfect but was is in life?

author: PlainName
author_rank: Super Contributor
timestamp: October 19, 2022, 07:47:38 am
content: Quote from: brucehoult on October 19, 2022, 01:38:52 amI am slightly puzzled. You say that a live system can be imaged BUT there is a problem doing work on it while it's being imaged. That's the issue that I've been banging on about, and which there has been no resolution when I've previously broached it. (Anyone interested can search for those threads and see for themselves that, once it gets to actual recommendations, nothing turns up).VSS is a relatively recent thing for Windows, but many backup imaging programs managed just fine before that. I think the oldest one I can remember (and that I've used, and actually still use) isDriveSnapshot.(The website says it works on pretty much any Windows from W2K to W11, servers and workstations, and I can confirm that it images live W2K every night here. And restores - to transfer systems between phyiscal machines and/or VMs I simply restore backups rather than copy the existing system, because it's a useful check that the backups are actually restorable.)Perhaps you could recommend an actual application for creating images of a live system? One that can cope with diffs and multiple backup destinations? I'd be very grateful if you could solve my problem (and, no doubt, others would be grateful to be shot of me).

author: tggzzz
author_rank: Super Contributor
timestamp: October 19, 2022, 08:03:38 am
content: For the avoidance of doubt, PlainName == dunkemhigh.https://www.eevblog.com/forum/supporters-lounge/username/I wonder why he wanted to change his name.

author: PlainName
author_rank: Super Contributor
timestamp: October 19, 2022, 08:08:39 am
content: Quote from: Nominal Animal on October 19, 2022, 03:08:56 amAt the risk of upsetting Some People I won't quote everything. Or, indeed, anything! However, that doesn't mean I don't appreciate your entire post - it is very useful. And, based on that I think I shall modify my problem to be: "I can't do it on Linux". No doubt that will merely produce the response from some that I am lacking in moral fibre or something, but they will be missing that I am capable than most Windows users and they would just be reinforcing the view that Linux users are elitist.Regarding your application suggestions, I did look at bacula and similar but discounted it on the basis that it didn't fix the problem (multiple copies of the backup on different media) whilst adding another link in the chain of potential disaster bottlenecks (the server). Fine for an IT department but not me.Over the years I've used a number of different backup solutions, from tape to Timeshift clones via mirroring, etc. The two things I am sure about are:1. If it doesn't exist on a live system, it might well have gone to bit heaven.2. Any impediment greater than 'do it now' will prevent backups from being done.

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 19, 2022, 08:38:03 am
content: Quote from: PlainName on October 19, 2022, 07:47:38 amA human problem, not a technical problem.A snapshot file system image is an atomic snapshot. There is no guarantee that that snapshot is of a point in time that is preferable to us humans. If there is an application saving a file at that point in time, the snapshot will contain a half-modified snapshot of the file.This has nothing to do with any of the tools or underlying implementations of snapshotting. There is no technical problem or any kind of technical issue or slowdown or failure here; just a note that when doing filesystem-wide snapshots, the moment one takes the snapshot is significant.One way to get around this, is monitor accesses to the filesystem both before and after the snapshot.For the entire filesystem, there are a few different options. The kernel does have interfaces that the monitoring/backup process can use to find out which files are currently open for writing in the target filesystem. This gives an easy metric, if the snapshotter uses two temporary snapshots, and always keeps the "more useful for humans one" and discards the other, considering write accesses before and after the snapshot, and uses the best surviving one as "the" current snapshot. Or you can use some other metric.When backing up individual files instead of an entire filesystem, the same problem exists for individual files. In Linux, if the user running the monitoring/backup process is either superuser or the same as the owner of the files, the monitoring/backup process can use a feature calledfile leasesto check if that particular file is being accessed right now, or in the near future. A write lease only succeeds if no other process has the file open or memory-mapped. Also, when the process has a write lease, then any attempt to open the file will be delayed (not blocked, only delayed) for up to a time limit, or when the lease owner releases the lease. The open attempt will be notified using a POSIX signal.In no case is the snapshot "denied" because of filesystem accesses. The entire thing here is about how to ensure the snapshot is maximally useful to us humans, and not some technical corner case when snapshotting fails –– be that for entire filesystem or specific files.One reason cross-OS backup utilities don't get this right, is because different operating systems tend to have different approaches to this. So, a single approach or algorithm won't work equally well on all systems, and instead you need to understand each OS and its quirks separately. I'm convinced that trying to port any of my own backup stuff –– except the plain "tar this subtree every day at this time, and copy to this directory; keeping only the last N tarballs" –– to a non-Linux OS would lead me to desperation.Quote from: PlainName on October 19, 2022, 08:08:39 amRight; I definitely have no issue with that. I can even add to that that "Doing it in Linux requires quite in-depth knowledge and initial investment of time and effort".I do admit that one facet of this is very frustrating to me: because I've used Linux every single day in all kinds of environments from embedded to desktops to servers to HPC clusters, and I find all this as natural and comfy as well-fitting old slippers, it is difficult for me to try and find solutions from the perspective of an user who does not have that kind of experience. That is, I myself am too removed from their context to give advice I'd really be happy with.Moreover, whenever I've optimized others' workflows, I always start by observing their current workflow, tasks, and approaches. I need tounderstandthe problem or task or workflow first, before I start making any kind of changes willy-nilly.Online, such observation is impossible. Even when sharing the desktop, all the important cues like exactly what the user likes or dislikes (as opposed to what theythinkthey like and dislike, the former telegraphed by their facial expressions and body language) are lost. Only a tiny fraction of the necessary information gets conveyed.

author: nctnico
author_rank: Super Contributor
timestamp: October 19, 2022, 09:23:01 am
content: Quote from: PlainName on October 19, 2022, 08:08:39 amYou have to be aware that Windows and Linux are different. You may find a lot of utilities are 'missing' from Linux / Unix because you are used to needing a seperate GUI based utility for every function on Windows. Linux is different. In my case: I have been looking for a Linux utility to create an image from a CD / DVD. It doesn't exist because you do this with 1 simple command on the command line. The same for backup, file replacements, etc; there are oodles of command line tools that can do the most crazy things from a simple bash script. An example: a couple of months ago I created a bash script that takes some encryption keys, processes them and creates a new file which is signed using the keys. There is a lot of flexibility in that approach but as a Windows user, it did take me a lot of getting used to work that way.

author: brucehoult
author_rank: Super Contributor
timestamp: October 19, 2022, 10:30:53 am
content: Quote from: PlainName on October 19, 2022, 07:47:38 amI see DriveSnapShot claims:QuotePlease explain how it does that on a file system (e.g. on Win 2k) that does not support snapshotting?

author: PlainName
author_rank: Super Contributor
timestamp: October 19, 2022, 10:36:07 am
content: Quote from: brucehoult on October 19, 2022, 10:30:53 amDon't know. It just does. If you need to know the details then I'm afraid you'll have to ask the chappy that wrote it - I am just a happy user that knows it works.

author: Picuino
author_rank: Super Contributor
timestamp: October 19, 2022, 11:01:33 am
content: Perhaps this thread should be split to deal with the new topic in a separate thread and not mix both topics.

author: brucehoult
author_rank: Super Contributor
timestamp: October 19, 2022, 11:09:57 am
content: Quote from: PlainName on October 19, 2022, 10:36:07 amHow do you know it works? Maybe it only works 99.9% of the time? That's my bet.I expect some of the other participants in this thread are considerably more qualified in this matter than the chappy that wrote it.

author: PlainName
author_rank: Super Contributor
timestamp: October 19, 2022, 11:18:41 am
content: Quote from: Picuino on October 19, 2022, 11:01:33 amI think we are probably done on the diversion. Sorry!

author: PlainName
author_rank: Super Contributor
timestamp: October 19, 2022, 11:24:01 am
content: QuoteActual disaster recovery where files expected to be trash with other backup means (and even from just an unexpected power off) were fine. I have never not had a successful recovery in that regard.

author: slugrustle
author_rank: Frequent Contributor
timestamp: October 19, 2022, 05:08:00 pm
content: Quote from: Picuino on October 19, 2022, 11:01:33 amThis is a nice sentiment, but I think that "side" discussion has the approximate momentum of a freight train. I'd bet that >50% of the content on this forum is nerd arguments.

author: Picuino
author_rank: Super Contributor
timestamp: October 19, 2022, 05:36:40 pm
content: AVRShellhttps://github.com/rahra/avrshellAVR shell is the ultimate development, debugging and education system for AVR-based microcontrollers.AVR Shell is tiny self-contained multi-tasking OS to be executed on 8 bit AVR cores such as the Atmega328p which is found on popular Arduino boards such as the Arduino Uno or Arduino Nano.The purpose of AVRShell is mainly to support your hardware development and or testing your software or for educational purposes.The AVR Shell provides a tiny command line interface which enables you to read and write the AVR memory, including all registers and the program memory, by using simple commands.ConnectingSimply connect to your Arduino with a serial terminal program such as minicom. Just run minicom -D /dev/ttyACM0 -o -b 9600 -w.CommandsThe following commands are implemented yet.in <reg_addr> ... Works like the in instruction but is implemented with memory mapped instructions (ldd).out <reg_addr> <value>. ... Works like the out instruction.sbi <reg_addr> <bit> ...... Set bit (0-7) in IO register.cbi <reg_addr> <bit> ...... Clear bit (0-7) in IO register.dump <addr>  ......... Dumps len (default = 512) bytes of the memory.pdump <addr>  ........ Dumps len (default = 512) bytes of the program memory.edump <addr>  ........ Dumps len (default = 512) bytes of the EEPROM memory.ste <addr> <byte> ......... Write byte to EEPROM at address addr.cpu ....................... Output CPU information, such as fuse bits, lock bits and signature.uptime .................... Show system uptime ticks since last reset.run <pid> ................. Run process pid.stop <pid> ................ Stop process pid.new <address> ............. Create new process with start routine at address.ps ........................ List processes, with pid, current stack pointer, and state. States are defined in process.h.All these commands are implemented using ld, lpm, and st.

author: PlainName
author_rank: Super Contributor
timestamp: October 19, 2022, 05:59:05 pm
content: For a non-language CLI, isargtable3beyond considering? I'm using it on ESP to provide remote command access (typically to enable/disable/config stuff like a top clone), so although it probably won't fit an ATTiny it doesn't need full-fat Linux either.

author: NorthGuy
author_rank: Super Contributor
timestamp: October 20, 2022, 03:25:28 am
content: Quote from: Picuino on October 19, 2022, 05:36:40 pmHave you ever heard of debuggers?

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 20, 2022, 03:33:51 am
content: Quote from: NorthGuy on October 20, 2022, 03:25:28 amYou really don't understand how these devices can be used, do you? Not everything is about finding out why your crappy code does not work like you thought it would.

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 20, 2022, 03:38:49 am
content: Quote from: PlainName on October 19, 2022, 05:59:05 pmWhat I described above in my overly verbose way, was pretty much about how to do that in a way that will fit in an 8-bitter. ATTiny is on the border; 8k of Flash is not much. You'll need to convert incoming strings to tokens (perfect hashing or similar), or numeric values within the input parser, but otherwise it works very much the same way. If you use GCC or Clang as the compiler, you can even sprinkle the command definitions in the compilation units where the functionality is defined, no need for a single array either. (Uses ELF sections.)

author: brucehoult
author_rank: Super Contributor
timestamp: October 20, 2022, 07:33:48 am
content: Quote from: Nominal Animal on October 20, 2022, 03:33:51 amKeep calm, mate, you're not drunk'n'high, I hope.

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 20, 2022, 08:48:57 am
content: Quote from: brucehoult on October 20, 2022, 07:33:48 amIt's just that we already established pages ago that the really useful use cases are not debugging the same firmware, but using such simple command languages to explore, test, and debug stuffconnected tothe MCU running such firmware. AVRShell is not used to debug the firmware itself, but as a simple programmable tool for debugging other things by connecting them to an AVR running AVRShell.Having then someone do a drive-by "ha, you're stupid, don't you know debuggers exist?" post just makes me a bit prickly. (Feel free to drop the -ly.)

author: nctnico
author_rank: Super Contributor
timestamp: October 20, 2022, 09:08:18 am
content: Quote from: Nominal Animal on October 20, 2022, 08:48:57 amI disagree; your proposed use is way too limited. I have been using command line interfaces for decades to debug/verify software running on microcontrollers. Not only hardware related modules, but also higher layers. It has proven extremely useful; I can ask a client to connect a serial interface and run some commands to query the status or do this myself over a remote connection.

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 20, 2022, 09:34:17 am
content: Quote from: nctnico on October 20, 2022, 09:08:18 amIf I had written "... not debugging the same firmware, butfor exampleusing such simple ..." –– as I should have –– would you still disagree?As I see it, you've not used it as a debugger replacement, but an interface to access the state of the running system. This is not nitpicking: debuggers are in control of the execution on the target device, and have to be, to be useful. This, however, is what I like to call "a query-response interface": a facility for accessing thestateof the system in a very limited, but still very useful, fashion.

author: nctnico
author_rank: Super Contributor
timestamp: October 20, 2022, 10:23:30 am
content: IMHO you should see a debugger and command line interface as complementary. Both are equally usefull. In addition a command line interface makes field service much easier. Some of my products can run an extensive self test that tell me exactly what is wrong. The next step is that the customer can send me a particular module for repair. It saves me time to visit the customer and spend time on diagnostics

author: PlainName
author_rank: Super Contributor
timestamp: October 20, 2022, 10:39:03 am
content: Quote from: Nominal Animal on October 20, 2022, 03:38:49 amI missed the detail, sorry. For small micros I tend to use single-character commands and dispense with parameters, so they're effectively already tokenized

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 20, 2022, 02:59:35 pm
content: Quote from: nctnico on October 20, 2022, 10:23:30 amAgreed; well said.Quote from: PlainName on October 20, 2022, 10:39:03 amYep, that works.And likeslugrustle described in #128, you can also switch to binary, to a bytecode interpreter. That can be extremely frugal wrt. RAM and ROM/flash.

author: NorthGuy
author_rank: Super Contributor
timestamp: October 20, 2022, 04:27:26 pm
content: Quote from: Nominal Animal on October 20, 2022, 08:48:57 amJust a little bit?It doesn't matter what you thing debuggers are designed for. You can use them to read MCU registers, write MCU registers, dump memory, switch PC to the desired position, run code snippets, stop the code at the desired point etc - all these things that you want to do with "AVRShell". How can you not see the similarity?

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 20, 2022, 04:52:43 pm
content: Quote from: NorthGuy on October 20, 2022, 04:27:26 pmAnd you still can't fathom any other use case.You didn't notice that it implements multitasking, did you? Go have a look atsrc/timer.Sandsrc/process.S. It's a cute little educational tool, helping new AVR programmers explore more than just the Arduino-like "loop plus interrupts".Again, the AVRShell commands are used to access MCUstate, without stopping everything running in it. It is a complement to a debugger; more like a playground than a crude debugger you think it is.

author: Picuino
author_rank: Super Contributor
timestamp: October 20, 2022, 04:54:47 pm
content: In addition, you can automate the sending of commands from the PC with a programming language, which you cannot do with a debugger.

author: NorthGuy
author_rank: Super Contributor
timestamp: October 20, 2022, 05:43:07 pm
content: Quote from: Nominal Animal on October 20, 2022, 04:52:43 pmI am not talking about use cases. I'm talking about capabilities, which answer the question "what can you do?". The use cases answer the question "What are you trying to accomplish?" If you really have a use case where you had to accomplish something on a tiny microcontroller and you used an interpreted language to do this faster and better, post it here. Then we'll see how it helped and perhaps how it could've been done differently.Quote from: Nominal Animal on October 20, 2022, 04:52:43 pmIf it accept commands and runs at the same time, it's got to be multitasked.If you're talking about user thread, C and certainly do this, with RTOS, or otherwise. I don't see why there's a need for interactive shell for this.Quote from: Nominal Animal on October 20, 2022, 04:52:43 pmAs to accessing MCU memory (be it flash, RAM, or hardware registers), many debuggers allow you watching variables without halting, but not all the chips are able to do this. For those chips which can the debugger is simply connected to the data bus. Thus, aside of bus contention, there's no overhead. If there's no such facilities, any debugger can halt-read-run with less overhead than AVRShell. Some debuggers support breakpoints at data reads/writes, which minimizes overhead even further.Most importantly, the resource use is minimal, or not at all. Say, tiny PIC16F15213 with 2k flash and 256 bytes of RAM. You can debug it interactively without using any resources. You don't even need to write a sensible program. You can start the debugger and start exploring. Say, enable PWM, write to the duty cycle register and watch how this affects the things outside.

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 20, 2022, 06:03:23 pm
content: Quote from: NorthGuy on October 20, 2022, 05:43:07 pmSo, where is your debugger that provides multitasking capabilities on the target?Do consider going back to your day job, torturing customers as a tier-0 tech support.

author: NorthGuy
author_rank: Super Contributor
timestamp: October 20, 2022, 06:03:42 pm
content: Quote from: Picuino on October 20, 2022, 04:54:47 pmYou certainly can. Most debuggers have command line interfaces, so you can invoke it from your program and make it execute any commands you like.But I don't think this is needed. If you want something on MCU with the interface, you just create the interface first, then use it.Few years ago I had to make a machine which had 7 stepper motors which had to be controlled from PC. I created UART command interface between the MCU and PC. Then I buit a GUI application (could've been CLI as well) which let me issue commands - simple ones, such as moving a single motor, or more complex ones which move several motors in synch without bumping into each other. This let me explore how the motors move and adjust things. I kept modifying the PC program adding more and more complex motions until I got the snippet which would run the machine unattended. I did it all on the PC. With compiled languages. The MCU only had to decode UART commands and send signals to motors. Some may think this would be a good case for interactive languages, but this is not. Moreover, motors need to receive pulses at exact times. A bloaty interpretative language would mess everything up.

author: NorthGuy
author_rank: Super Contributor
timestamp: October 20, 2022, 06:20:08 pm
content: Quote from: Nominal Animal on October 20, 2022, 06:03:23 pmDebugger itself is a piece of hardware (sometimes part of CPU, sometimes external) which does its job independently of the CPU. It watchs for breakpoints, accept interrupts or commands from the debugging software on PC etc. This activity is parallel to CPU execution and doesn't affect it until CPU halts.Debugger doesn't provide multitasking capabilities on the target. The firmware can provide multitasking capabilities if needed. The MCUs also have peripheral modules which are independent on CPU. So, while the CPU is halted by the debugger they can continue doing their jobs, such as producing PWM etc.You refuse to provide a clear use case. It is quite impossible to discuss abstract things without any clue on how they relate to reality.

author: SiliconWizard
author_rank: Super Contributor
timestamp: October 20, 2022, 06:26:42 pm
content: Quote from: Picuino on October 20, 2022, 04:54:47 pmYes you can.

author: brucehoult
author_rank: Super Contributor
timestamp: October 20, 2022, 08:57:38 pm
content: Quote from: Picuino on October 20, 2022, 04:54:47 pmFunny, but that's exactly what OpenOCD, for example, does: scripts gdb to download and run code on the targetAs do any number of GUI wrappers for gdb, including elisp script in the emacs editor, to make gdb mode.

author: tggzzz
author_rank: Super Contributor
timestamp: October 20, 2022, 10:07:33 pm
content: Quote from: NorthGuy on October 20, 2022, 06:20:08 pm.Debuggers don't have to be hardware in addition to the CPU, and the early microprocessor ones didn't. They could set breakpoints and single step, but the major omission was that it was not possible to halt when a variable was accessed.

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 21, 2022, 03:26:41 am
content: Quote from: NorthGuy on October 20, 2022, 06:20:08 pmSo, your snarky comment about using a debugger instead of AVRShell was just utter shit, thrown at the wall and hoping nobody notices your stupidity.I'm sure your tier-0 telephone support shift is starting soon, so instead of wasting time trying to show what is actually possible to an idiot, I'll do us both a favour and ignore you, and go help e.g. students learning how unrelated tasks in a low-level 8-bit environment interact.I recommend everyone interacting with you do the same, because you obviously have zero idea about anything you comment. Just take a look at your posting history: nothing worth reading there either.

author: tggzzz
author_rank: Super Contributor
timestamp: October 21, 2022, 10:05:15 am
content: Quote from: Nominal Animal on October 21, 2022, 03:26:41 amI'm becoming cranky too. Too many posters appear (to me, naturally) that they would score poorly on a "critical thinking analysis" type test in a job interview.Tell tale: snipping context too heavily so that their point seems reasonablein isolation. Usually accompanied by a wail that "too much context makes posts too long, and I only removed unimportant stuff", or similar.I used not to like CTA type tests in job interviews; my result of being in the "top 3%" seemed trite and somewhat surprising. I'm coming to understand what they do trap out

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 21, 2022, 01:34:00 pm
content: Quote from: brucehoult on October 20, 2022, 08:57:38 pmgdb is a pretty powerful debugger. I've used its Python (gasp!) interface for pretty-printing and complex data structure tracking and visualization; it all startedhere.Quote from: tggzzz on October 21, 2022, 10:05:15 amI don't know if you think I belong to that set, but nevertheless, I do agree.A quick recap of how I saw the most interesting bits in this thread:nctnico first described the rough idea of using a query-response interface or a scripting language for the high-level logic only, in#5. Picuino's post#6describes their initial intended use cases –– roughly,learning and exploration, not to avoid using a proper debugger. In#17rstofer outlined how they used a higher-level language (TinyBasic) for "testing" 8085 projects.Next, tellurium brought up doing initial embedded environment on a desktop machine in#24, with others (nctnico in#25, tggzzz in#26, etc.) describing positive experiences doing that. In#30, I roughly sketch how one can use Gtk in Linux for much easier embedded user interface development.SiliconWizard pointed out in#38that a better approach would be to look at onesneeds, and starting from there instead.In#52, nctnico suggested a command-line interface for Picuino's use cases.In#64, I mentioned that my own use cases vary from experimentation to debugging. I didn't think to mention that I don't use it to debug the code on the microcontroller itself, but other things connected to the microcontroller. (I once even used a pair of Teensys connected via UART (very short wires), so that I could use one to "pose" as a completely different USB device (a vinyl cutter), "playing back" captured USB data, but under the control of the other Teensy, and that other one controller by my program. I could have used a kernel driver to skip the entire USB hardware loop, but this was easier.)I spent most of the message trying to show that one doesn't really need a full programming language for this kind of stuff, as a very simple query-response or command line interpreter will do just fine.In#86, I show the key snippets I use, and finally describe my own uses, like experimenting with external actuator parameters for optimal responses, plus this:Quote from: Nominal Animal on October 16, 2022, 03:21:28 amOthers described similar use cases (nctnico#101, slugrustle#128and#136): that use case pattern seems to be quite old and well known.In#161, Picuino brought up AVRShell. I consider it an excellenteducationaltool, for someone learning embedded development. The way you can easily add your own C code "programs" to be multitasked, and with a few commands explore the entire state while the device is running, is definitely useful as a learning tool.The key here is possibly nctnico's post#170:Quote from: nctnico on October 20, 2022, 10:23:30 amExactly. I myself use the query-response/command interfaces in microcontrollers I use as tools. I can see very interesting educational/teaching possibilities with AVRShell. Having a simple, robust way to check the internal state of a running system (already in production) is certainly useful. These are all use cases where you really don't want to have to use a debugger instead. (And in cases like production systems, it may not be at all feasible to provide a debugger to the end user anyway.)Getting bogged down with someone not understanding and utterly ignoring all other uses except for debugging code in a product, is .. well, so frustrating that I ended up writing this post.

author: NorthGuy
author_rank: Super Contributor
timestamp: October 21, 2022, 02:01:07 pm
content: Quote from: Nominal Animal on October 21, 2022, 03:26:41 amUsing for what? You keep leaving out the purpose (which, in my mind, is exploring MCU and hardware connected to the MCU). Our conversation goes approximately like this:- Use pliers to hammer nails- Have you ever heard of a hummer?- But does your hummer have moving part?- No- Then you cannot use hummer instead of pliersAlso, if you could skip profanities, I think it would be good for everybody, in the first place for you.

author: tggzzz
author_rank: Super Contributor
timestamp: October 21, 2022, 02:05:12 pm
content: Quote from: Nominal Animal on October 21, 2022, 01:34:00 pmIt doesn't matter to me either way. I prefer "cranky (or worse) plus correct" over "polite but wrong or time wasting" any day.QuoteDebuggers are inherently dangerous, especially when the operator is tired or or under pressure from above or suffering from Dunning-Krueger syndrome.The halfway camp to a debugger is a decent logging system. If done well it can directly indicate the problem to be fixed, or show the problem is in another company's equipment, and/or prevent law suits.QuoteJust so.

author: tggzzz
author_rank: Super Contributor
timestamp: October 21, 2022, 02:07:07 pm
content: Quote from: NorthGuy on October 21, 2022, 02:01:07 pmI have no doubt you believe all that.I suggest you stop digging.

author: NorthGuy
author_rank: Super Contributor
timestamp: October 21, 2022, 03:33:38 pm
content: Quote from: Nominal Animal on October 21, 2022, 01:34:00 pmThat's not true. Modern debuggers do not affect execution of the program in any way until it halts for whatever reason. If not halted, it runs absolutely normal. Often to the point that the program cannot even determine that it runs under the debugger.If you want to explore actuator parameters, you don't really need anything, even a debugger. If you actuator is powered by a stepper motor you probably need to determine maximum speed, as well as acceleration and deceleration profiles at various loads. So, you just program a simple motion, program it to the chip and run, which takes about a second or less. The motor moves (correctly or incorrectly), you vary the load, run the program again, observe the differences, measure things. Then you change your problem a bit (say change the acceleration profile), and see how the results have changed. Setting up a debugger, or some system of interactive commands is just a waste in this case. Re-compilng is quick enough and you don't need to go though any hurdles to avoid it.Sensors are all different. There are some where everything is pure software. For example DS18S20 were very common 15 years ago. To read the sensor you had to bit-bang rather lengthy sequence on a single wire. This is best explored with a scope or LA which will show you what sequence you're actually producing. Other sensors are just ADC, which are best explored with periodic "printf".As you can see, even the debugger is not that much needed for the exploration.Unlike the debugger, if you use things as AVRShell or interpreted language, this adds unnecessary bloat. In the links you posted, you can see that AVRShell produces timer interrupts which will disturb timing of whatever else you're doing. For example, this may change timing of your acceleration profile with the actuator, or it may wreck the timing of your bit-banging with DS18S20. And what do you get in return? Nothing.

author: PlainName
author_rank: Super Contributor
timestamp: October 21, 2022, 04:03:02 pm
content: QuoteIn the context of a MCU I would question whether bloat is important. Sure, on something like a PC or general purpose kit it is, but, on a single-purpose MCU, what would take the place of the bloat? Nothing. Unless you're actually running out of resources it is free (apart from the development time to implement).Also worth considering that if you've got it you can use it or not, but if you haven't then you can't.

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 22, 2022, 05:42:02 am
content: I'm a toolmaker at heart. I like to make small gadgets –– both software and hardware –– to solve problems, but also to explore their solution space, and the various approaches to attacking a problem. If you've never used FFT to implement scalar multiplication, you probably find this odd. In practice, it means no problem is unsolvable to me: it is only a matter of available time and resources.I also find learning –– supporting others trying to learn –– more important than any current abilities. Because humans learn in different ways, you cannot teach effectively by having a single lesson that fits everybody. You need to vary the approach to suit each individual, and their way of thinking.Among developers, there is an axis of variance, where most people fit cluster at one or the other extreme. This axis is akin toculture, but in the domain of development tool use.Some developers demand an integrated environment, because they feel they are more productive in one, and it feels safe and supported. They prefer a close vendor relationship, because it minimises development effort, cost, and cognitive load.Other developers want modularity, and the ability to choose their tools. They are naturally drawn to theUnix philosophy, as instead of using each tool for their designated purpose, they combine multiple tools to achieve any specific task. They find any 1:1 mapping between task and tool limiting.Here, we have NorthGuy being a textbook case of someone who has never undestood, and does not understand any approach outside their own. They have never taken a critical look at their own workflows, and measured their own total productivity when using different approaches. In#185, I listed various scenarios where using a debugger is really not suitable, as described by myself and others (including end-user operational state verification, running system parameter optimization, and making tools for investigating other systems and components). Yet, the likes of Northguy still demand use cases,because they do not understand, and cannot even imagine themselves doing any of this; they just skip over these parts as if they weren't even discussed.The reason I called their post#163, "utter shit",Quote from: NorthGuy on October 20, 2022, 03:25:28 amis because AVRShell is not a debugger. It is an environment for learners and hobbyists to explore their own C code inside a "shell" that provides multitasking (a fixed time slice scheduler) and simple commands and state examination while everything is still running using a serial connection. It is literally described as so in the first sentence in Picuino's post#161, just after the title and the link to the post.NorthGuy's post is a typical snarky remark by someone who cannot see anything beyond their own current use patterns, being an indicator of their own limitations and downsides than anything else. I'm sad to see that a few members here seem to agree with him. Not because it has any impact on myself, but because they really cannot see how they're limiting themselves for no reason. NorthGuy's reference in#186to using "pliers to hammer nails" is just hilarious, with zero self-awareness.Perfect example of somebody who simply cannot see problems as anything but nails, and wondering why anybody would use any tool except a hammer.

author: tggzzz
author_rank: Super Contributor
timestamp: October 22, 2022, 09:43:00 am
content: A few random comments on your interesting post...Quote from: Nominal Animal on October 22, 2022, 05:42:02 amThat pretty much describes me too.I'll add that two other things give me disproportionate pleasure:allowing people to realiseand dothings they either didn't believe or didn't believe they could do. Watching people radically expand their weltanschauung and capabilities is a true delight. Anybody with childrenoughtto realise thatallowing (or forcing sometimes) people to see that the world isn't black and white, it is all shades of grey - and that apparently different things are closely related. My canonical example is there is no simple distinction between hardware and softwareQuoteIt feels a little worse than that to me: someone who cannot evenperceivethere are alternative viewpoints, and is notinterestedin expanding their horizons.QuoteSpot on. You've hit the nail on the head (with the stilson wrench).

author: nctnico
author_rank: Super Contributor
timestamp: October 22, 2022, 10:34:22 am
content: Quote from: Nominal Animal on October 22, 2022, 05:42:02 amYou have to keep in mind that everyone has limitations. But that doesn't mean people can't be productive. You just need to make sure not to overload a person with a task / toolset that is beyond the particular person's abilities. I've been part of projects where people got a task that was way over their heads and subsequently failed or just quit. That made me very sad because I had already warned about that.

author: MK14
author_rank: Super Contributor
timestamp: October 22, 2022, 10:46:32 am
content: Quote from: nctnico on October 22, 2022, 10:34:22 amIt is not just that. If someone/people are overloaded like that. I suspect, their efficiency and productivity, will become significantly worse. Which is an inefficiency waste of time and other often expensive resources.E.g. Forcing everyone to learn a new programming language, because it is the hottest new one available. Instead of sticking to the one, the whole department knows and is use to.

author: tggzzz
author_rank: Super Contributor
timestamp: October 22, 2022, 10:59:00 am
content: Quote from: MK14 on October 22, 2022, 10:46:32 amI strongly agree with the concept of not jumping on the new hot programming language of the month. Several times I've deliberately avoided learning some new languages because they offered insufficient new capabilities or advantages. Prime examples that spring to mind are C++, Delphi, C#. (But primenon-examples are Algol, one assembler (or microcode!), C, Smalltalk, Objective-C, Java, Mathematica, Matlab, analogue hardware, digital hardware, VHDL)However, that implies understanding the concepts of different languages and paradigms sufficiently well that you can evaluate what benefits they might offer to the problem in hand. Too few people do that.Too many people choose not to bother to do that.Right tool for the job. Horses for courses. Etc

author: MK14
author_rank: Super Contributor
timestamp: October 22, 2022, 11:22:24 am
content: Quote from: tggzzz on October 22, 2022, 10:59:00 amAlthough, the new methods/tools. Such as a decent debugger, for the tiny microcontroller. May, supposedly offer various advantages in theory. Which themselves, would improve productivity.They can often involve a significant learning curve to climb, in order to make them as productive, as the previous method(s), used in the same circumstances.E.g. If a person is currently using AVRshell, and is getting good results. But coping with its limitations, compared to bigger and more expensive tools (if available and they exist, for the specific tiny MCU used).It might be quicker and more productive, to stick with the existing tools, warts and all, because it delays or removes a possible time sink, with learning how to use complicated new tools/languages etc.I suppose the best solution. Is to make an informed/smart decision at the time. As to which approach is best, under the circumstances with time/budget and other constraints needing to be taken into account.Sometimes the true cost of changing to supposedly better tools/languages etc. Is not fully understood. E.g. Various homegrown utilities have to be rewritten, hardware changed, courses paid for. New books and other stuff, may need to be obtained. Some usage and other documentation guides (written by that person or department), may need to be revised or rewritten.Also, current project constraint(s), especially as regards deliverables and their timescales. May make it impractical to change tools/languages/equipment/manpower.

author: tggzzz
author_rank: Super Contributor
timestamp: October 22, 2022, 03:08:34 pm
content: Quote from: MK14 on October 22, 2022, 11:22:24 amOh, yes indeed.A key skill is being able toquicklydetemine what it isn't worth spending too much time learning. That leaves (too little) time to learn what really is worth learning.As for "right tool for the job", there are many factors in that, some technical, some financial, some policital, some cultural. And I'm sure there are others not covered by that!

author: MK14
author_rank: Super Contributor
timestamp: October 22, 2022, 03:35:36 pm
content: Quote from: tggzzz on October 22, 2022, 03:08:34 pmAlso, even if you have the tools available, and are efficient at using them. Unless it REALLY needs it, because of an extremely tricky bug or bugs. It can save a lot of time and hassle, even setting up and using such equipment.E.g. A multimeter, might take seconds, to grab and probe a circuit, to get the voltage across it. But an oscilloscope, can take considerably longer, to get productive with it.Edit: Made significantly shorter, but a bit more to the point.

author: NorthGuy
author_rank: Super Contributor
timestamp: October 22, 2022, 09:11:45 pm
content: Quote from: Nominal Animal on October 21, 2022, 01:34:00 pmI did that too. With PIC32MK which has two USB modules and UART interface as well. So, let's stick to that example. I didn't use any debuggers or interpreted languages. I don't understand what is the relation to AVRShell.A device which uses command protocol to inspect functioning of the CPU (including memory) is called a debugger. Many MCUs have hardware debugging interface which provides means to access CPU data. The debugger on PC uses command interface (although it's usually in binary form rather than ASCII format) to access MCU's debugging hardware. It then exposes CLI or GUI interface to the user. Debugging commands can also be transferred to remote locations (e.g. across Internet).It is possible to create your own program which provides debugging functionality (such as AVRShell), even on an MCU where there's no hardware support for the debugger. But such implementations will always be inferior to hardware debugger.If you simply need a command interface to your device, such as transferring commands which govern USB communications in the above example, you don't need neither debugger, nor AVRShell. In the simplest form, a single C switch statement to perform different actions in response to incoming UART bytes. How in the world AVRShell may help here?

author: tggzzz
author_rank: Super Contributor
timestamp: October 22, 2022, 10:00:03 pm
content: Quote from: NorthGuy on October 22, 2022, 09:11:45 pmit is always worth keeping this quote in mind; there's a reason it is still remembered 150 years after it was written...When I use a word,’ Humpty Dumpty said in rather a scornful tone, ‘it means just what I choose it to mean — neither more nor less.’ ’The question is,’ said Alice, ‘whether you can make words mean so many different things.’ ’The question is,’ said Humpty Dumpty, ‘which is to be master — that’s all.