primitive serial CLI / Console for embedded systems - Ideas for implementation? - Page 1
Forum section: electronics / general

author: Warhawk
author_rank: Frequent Contributor
timestamp: April 29, 2024, 02:25:02 pm
content: Hi guys,I am wondering if any of you could provide some inspiration. I have a small test system for the internal lab use. The system aquires ceratain digital data and prints them to the PC over a serial port.I am going to implement a console/CLI. The interface shall be human readable but also needs to interact with python.Typical commands would be:Code:resetbuffer_size xxxx // 0-1024adc_filter x //off, type-1, type-2trigger... and many moreAs you can see, some commands come with parameters, some don't.I see that there are two possible methods. Using variable arguments and parse the string (A) or asking for the parameter separately (B).Code:(A.1)CMD>buffer_size 1024 <enter>(A.2)OK(A.3)CMD>(B.1)CMD>buffer_size <enter>(B.2)PRM>1024 <enter>(B.3)OK(B.4)CMD>PS: I do not care much about the memory footprint, portability, or speed. However, readability and ease of use are important for me. Somebody else, or I, should be able to understand and modify the code base in future :-) I generally need the KISS principle.What are your favorite options? Share links, github or just your ideas.I will start:https://dojofive.com/blog/embedded-command-line-interfaces-and-why-you-need-them/https://interrupt.memfault.com/blog/firmware-shellThanks in advance!

author: nctnico
author_rank: Super Contributor
timestamp: April 29, 2024, 02:46:18 pm
content: What works for me is a struct like this:struct TCommand{char *command;void (*func)(int argc, char **argv);char *helptext;};which is used to create an array with commands and help texts. I use a buffer which only receives text (>=32 <127). When a 13 is encountered, the input buffer is split by spaces, the space replaced by 0x00 to mark end of string and an array with pointers to each parameter is created. The first entry into the parameter array is the command. Iterate over the commands to find an match and call the callback function. Note that the callback function is exactly like the main function of a 'regular' C program.Every piece of embedded software I have made or worked on has a CLI and this has been proven to be an immensly useful feature for development, integration testing, post-production testing and field debugging (logging & fault finding).

author: Warhawk
author_rank: Frequent Contributor
timestamp: April 29, 2024, 02:56:53 pm
content: Quote from: nctnico on April 29, 2024, 02:46:18 pmGood tip for the ASCII 32 to 127. Thanks. I see that you use also var arguments. Any open source project of yours that you could share with us? I am also interested how do you do parsing. I am a hobby coder therefore quite an amateur.

author: nctnico
author_rank: Super Contributor
timestamp: April 29, 2024, 03:21:30 pm
content: Unfortunately no code to share. But what you need is a function which replaces spaces by 0x0 and puts a pointer to the beginning of each parameter in an array and keep track of the number of parameters found. Then a for loop which iterates through the list with commands and comparing each command to the first element of the parameter found.

author: Warhawk
author_rank: Frequent Contributor
timestamp: April 29, 2024, 03:21:58 pm
content: And one extra question.Does it make sense to optimize the CLI for VT100/Putty/miniterm type of communication (each character transmits immediately after the keypress) or for terminals that "wait for enter" (e.g. Termite) and send the whole line?

author: tooki
author_rank: Super Contributor
timestamp: April 29, 2024, 04:00:00 pm
content: Quote from: Warhawk on April 29, 2024, 02:25:02 pmThat is the code I started from to develop the CLI for a piece of lab equipment I built.The use case is practically identical to yours: human use for development, but ultimately controlled by Python scripting for experiment automation.In my case (a 6-channel capacitance meter built around Smartec’s UTI chip, with a PIC microcontroller to read the UTI’s weird period-modulated output and communicate over RS-422), I made some settings “modes” (verbose on/off, debug on/off, echo on/off, and fast/slow mode) that persist between measurements, while others are per-call, like the channel and number of measurements.For example, “read 1 20” takes 20 sequential readings of channel 1 and returns them as a list, while “avg 1 20” takes 20 sequential readings of channel 1 and returns a single averaged value. In debug mode, the result includes statistics. Verbose mode outputs the units (pF), extra human-friendly info, and line numbers on lists. The idea is that in automated operation, echo, debug, and verbose are off and the unit returns bare numbers that are easy to parse.I also added single-letter “aliases” of the commands, so that the two examples above can be typed as “r 1 20” and “a 1 20”, respectively.The mode commands accept a Boolean value (true/false, t/f, 1/0, and on/off are all accepted), but function as a toggle if no value is passed. This allows the control script to positivity set a known configuration before sending commands. One of the things the CLI parser returns is the number of arguments received (including the command itself), so you can easily make your command handler do different things depending on the number of arguments.Oh yeah, and I added backspace support (quite handy for us humans).

author: tooki
author_rank: Super Contributor
timestamp: April 29, 2024, 04:01:58 pm
content: Quote from: Warhawk on April 29, 2024, 03:21:58 pmDoes that matter?The important thing, IMHO, is that you parse the command as a whole, so that you can perform input sanitizing on the entire input as a whole.

author: djacobow
author_rank: Super Contributor
timestamp: April 29, 2024, 04:14:37 pm
content: The classic code to convert a string with spaces into an array of null-terminated strings uses strtok.Something like this:Code:uint8_t console_argumentize(const char *input_string, const char *argv[], uint32_t max_count) {uint8_t argc = 0;memset(argv, 0, sizeof(char *) * max_count);char *saveptr = 0;char *tok = strtok_r(input_string, " \t", &saveptr);while (tok != NULL && (argc < max_count)) {argv[argc++] = tok;tok = strtok_r(NULL, " \t", &saveptr);}return argc;}const char *argv;uint8_t argc = console_argumentize(input_string, argv, MAX_ARGS);Then, using the struct that nctnico provided, you implement some sort of find command that returns a pointer to the instance of the struct that matches argv (assuming argc is not zero and there is a match), and from that you use the function pointer therein to call the command function with the argc and argv you just created. Et voila!A fun thing I've done in my last project is to make a command that will take off the first argument and then redispatch to another command with the rest, thus letting me creating command with arbitrary nesting subcommands using the same framework.

author: nctnico
author_rank: Super Contributor
timestamp: April 29, 2024, 04:29:44 pm
content: Quote from: Warhawk on April 29, 2024, 03:21:58 pmIn my implementation I echo the incoming characters and support the use backspace for use with hyperterminal/putty/mincom/etc. Thats it. I find the single like terminals (like Cutecom) highly annoying to use because you need to switch between input fields all the time.

author: Nominal Animal
author_rank: Super Contributor
timestamp: April 29, 2024, 11:37:41 pm
content: Quote from: nctnico on April 29, 2024, 02:46:18 pmMe too; I recommend this approach.Apologies for the long post that follows; feel free to ignore. Some might find the ideas in it useful, though.On embedded/microcontrollers, I use a buffer to store the string data, so that it is trivial to support command editing via backspace (ASCII 8, BS) and so on.When a newline (NUL, LF, CR, or any combination) is received, I process the command, into something likeCode:#include <stddef.h>#include <stdint.h>#ifndef MAX_ARGS#define MAX_ARGS 16#endifunsigned char *arg_ptr;uint32_t arg_hlen;unsigned char args;Thehlenis a combination of hash and length, with length in least significant bits, and the DJB2 xor hash variant in high bits:Code:#ifndef LEN_BITS#define LEN_BITS 8#endif#define LEN_MASK ((1 << LEN_BITS) - 1)uint32_t hlen(unsigned char *const ptr) {// Empty strings yield zeroif (!ptr || !*ptr)return 0;size_t len = 0;uint32_t result = 5381;while (ptr)result = (result * 33) ^ (uint32_t)(ptr[len++]);return (result << LEN_BITS) | (len & LEN_MASK);}but they are calculated when the command buffer is tokenized. Each token will be terminated with NULs ('\0') by replacing the whitespace with NULs.For defining commands, I use a very similar structure as nctnico:Code:typedef __attribute__((aligned (sizeof (void *)))) struct {const unsigned char *cmd;uint32_t hlen;int (*func)(int, unsigned char *, uint32_t *);const unsigned char *help;} command_definition;and all command structures and their strings will be stored in Flash. The return value offunc()varies; sometimes I useconst unsigned char *, with the function either returning NULL for OK, or an error string. It also varies whether I pass thearg_as parameters tofunc()or not. (On AVR, ARM, and x86_64 architectures I do prefer to pass as parameters, as they can pass up to five scalar parameters in registers.)The reason for the separate typedef is that it allows thealignedattribute to set the alignment exactly. The last member is a pointer, to ensuresizeof (command definition)is a multiple of pointer size too.Because I use ELF-based toolchains (gcc, clang), I often use a dedicated section via a preprocessor macro (and standard linker script section start and end symbols) to collect all command structures into a linear array:Code:#define DEFINE_COMMAND(_var, _cmd, _hlen, _func, _help) \__attribute__((used, section ("cmds"))) \static const command_definition _var = { \.cmd = _cmd, \.hlen = _hlen, \.func = _func, \.help = _help \}The linker script exposes a symbol__start_cmdsat the start of the combined array, and__stop_cmds, so that you can useCode:extern const command_definition __start_cmds[];extern const command_definition __stop_cmds[];#define CMDS ((size_t)(__stop_cmds - __start_cmds))#define CMD(i) (__start_cmds)Thing is, the linker will combine allDEFINE_COMMAND()statements, even in completely different source files (as long as they are all linked to the same binary), into a single array this way. If you have different configurations, where some source files are included or dropped from the binary, this makes it very easy to control whether commands related to those source files are available or not.To find a matching command, you scan through theCMDs, checking if thehlenmatches. If it does, you do a string compare. This way, it is very fast to find the actual command even if you have a few dozen of them.A few years ago, I wrotethis RPN calculator example(at StackOverflow) to run in Linux as an example of how to use the ELF section mechanism. Basically, it implements a simple reverse Polish notation calculator, with operators (functions/commands) implemented in separate files. Just by selecting which files are linked in to the calculator, you select which operators are available.GCC and Clang generate the__start_sectionand__stop_sectionsymbols automatically. For other compilers, you need to edit the linker script to define the symbols.My projects often have a set of variables that I want to modify. (Sometimes the interface can only modify these variables, in which case I don't have a command interface at all per se, just an interface that acceptsvarname?to query a variable by name,?varnameto describe a variable, andvarname=valueto set a variable, with whitespace around=and?ignored.)These use the same basic logic, except with a different structure, definition macro, and section name:Code:typedef __attribute__((aligned (sizeof (void *)))) struct {void *ref;const unsigned char *name;const unsigned char *help;uint32_t hlen;uint_fast16_t type;void *limits;} variable_definition;#define DEFINE_VARIABLE(_refname, _var, _name, _help, _limits, _hlen, _type) \__attribute__((used, section ("vars"))) \static const variable_definition _refname = { \.ref = &(_var), \.name = _name, \.help = _help, \.limits = _limits, \.hlen = _hlen, \.type = _type \}extern const variable_definition __start_vars[];extern const variable_definition __stop_vars[];#define VARS ((size_t)(__stop_vars - __start_vars))#define VAR(i) (__start_vars)Thetypemember is basically an enum that dictates what therefpointer is cast to when accessing the variable. You can use any type you want for it, but it will take at least the same size as a pointer, because the entire structure needs to be aligned to pointer size.Thelimitsmember is eitherNULLor atype-dependent pointer to a structure defining the allowed range of values.These typically end up used via three commands:get,set, andhelp(ordescribe). Sometimes it can be useful to let the user define a few variables of their own (of some fixed type); you'd probably create those withletand delete withdel(orunset).You can then add support for arithmetic expressions using variables (and optionally functions), but I normally don't bother. It, too, starts by splitting the expression into lexical elements, but there are many ways to implement the parsing and evaluation of such expressions. In practice, you'd parse the expression into a stack of tokens, each token being either an operator (like+,-,*) or a value (either a number, or a reference to a variable). RPN is easiest, because it is simply a stack of values and operators. Theshunting yard algorithmis quite simple for normal math notation, but there are many otheroperator-precedence parsersyou might use.

author: Warhawk
author_rank: Frequent Contributor
timestamp: April 30, 2024, 09:11:29 am
content: Thank youeveryonefor great inputs.I now know better how to proceed. It seems that using variable arguments approach is preferred. Also, I should target putty type of interface rather than terminals that wait for enter before sending the string. Also, code examples from you are very helpful.

author: xvr
author_rank: Frequent Contributor
timestamp: April 30, 2024, 11:13:01 am
content: Just one note. It will be worth to make special command that will turn off symbols echo and editing capability - to use this CLI interface with Python. Python script do not require such echo, moreover, this echo will puzzle it

author: ejeffrey
author_rank: Super Contributor
timestamp: April 30, 2024, 12:03:26 pm
content: I too use basically the approach here.In my case, commands always fit on a line and are split by strtok_r. Responses are one of more lines of data, followed by "OK" or "ERROR nn" on a line by itself, using the C errno values as error codes. Those values can differ by platform so you will need a table from your microcontriller c library if you use more uncommon codes. I don't have any additional command prompt, I don't find it necessary for humans and is just extra for the program to skip.The data payload is in YAML syntax which is easy to generate from a simple program, easy enough to read manually and easily parsed python. YAML is not my favorite honestly but I find it nicer than JSON in this application. One advantage is having some block-quoting options for multiline strinfs which let you put blobs of text such as your help text with minimal escaping -- just adding an indent at the beginning of the line. I use this in a system where I have an MCU network and I need one controller to forward requests to another and communicate the response back. They both use exactly the same protocol, and the primary controller doesn't need to parse the responses at all, it just needs to block quote or and pass it on.

author: betocool
author_rank: Regular Contributor
timestamp: May 01, 2024, 01:11:39 pm
content: A leftfield answer here.I have been using Protobufs in C (nanopb) and python (pip install protobuf) since 2017 or 2018 successfully. You define the messages you want to use similarly to structs and then exchange them between micro and PC. The hardware is irrelevant, it works over any wire you can use as it's a serialised protocol.On the Python side I use a library called cmd, which basically acts as a CLI, very easy to implement. It supports autocompletion with tab and help and all that jazz without troubling the micro. On the micro side I use nanoPB to decode and encode messages.I have an example here:https://github.com/betocool-prog/Audio_Analyser_FW_OSCheck the "python" folder for the CLI implementation.Check "tasks/src/rmi.c" for the micro side of the implementation.The "build.sh" script in "protos" builds the c/h and .py headers for micro and PC respectively.Cheers,Alberto

author: NorthGuy
author_rank: Super Contributor
timestamp: May 01, 2024, 01:32:19 pm
content: If "ease of use" is the main concern, the vast majority of people will find GUI easier to work with than CLI. Communications with GUI based interface don't need to be human readable, therefore they're easier to implement and also will be more reliable.

author: T3sl4co1l
author_rank: Super Contributor
timestamp: May 01, 2024, 02:44:08 pm
content: For embedded use?I usually start with something based on this:https://github.com/T3sl4co1l/ReverbI've made minor updates over the years, but the basic skeleton remains; it's interactive, I can interrogate memory, I can write project-specific commands, I've got string functions (or use built-ins or libraries to taste, but, I'm one of those stubbornidiotspeople that writes their own routines so that's the form used here), and I can put in whatever other state I like.The main thing is it's not a full shell with grammar, pipes, etc., but I mean that would be pretty heavy weight for tiny embedded projects like this. You could find source to whatever, if you want a more programming-oriented interface -- Forth, BASIC, Lua, uPython, etc. Or Bash and etc., though I would guess those would be on the heavy-weight side (I mean not that uPython isn't , but, depends how many libraries you implement/include I guess, and more to the point, depends on MCU capability.....but, more to say, as far as having to implement or patch in the better part of a Linux backend/API to use Bash or etc., as such?).Anther downside is, since I never bother with flow control, and commands always spit out more characters than received (due to interactive echo + ANSI symbols + the command itself), buffer overflow is very easy to encounter, which limits automated use -- I can't simply copy-paste scripts into the terminal window for example. I can do a couple lines at a time, at least. If I ever wanted to write an interactive driver, I could just add buffer flush + delay to that. Or add a mode to disable echo, or interactive command entirely, but, it's simply not been enough of an issue to implement any of these so whatever.Tim

author: DavidAlfa
author_rank: Super Contributor
timestamp: May 01, 2024, 02:48:00 pm
content: Check this:https://github.com/imcbc/atom_console/