text command parsers? - Page 1
Forum section: electronics / general

author: westfw
author_rank: Super Contributor
timestamp: April 01, 2014, 04:02:03 pm
content: Is there a a defacto standard for interactively parsing text command lines in microcontroller code?(NOT the "unix command line"; there are plenty of libs for that. I'm talking about a program with a prompt that you type text commands at, like "set pin 13 = 1" or "clear uart 2" or "send westfw Hi Bill!"Preferably something Twenex-like (or cisco CLI), with context sensitive help, keyword completion, history and line editing. C or C++ preferred; (large-ish) microcontroller sized (I thought unix gettext might be a starting point, but it seems to focus on internationalization...)I could dust off CMU's "CCMD", but it's one of those 20y-old programs whose last edits were the addition of prototypes and modern C functions (vs K&R) and hacks for non-standard environments, and it was never tweaked to be modular on systems without various major features.

author: magetoo
author_rank: Frequent Contributor
timestamp: April 01, 2014, 04:53:23 pm
content: Don't know about anything commonly used in microcontrollers, but as a starting point: On the Unix side the usual suspects would bereadline(GNU GPL) oreditline(BSD). That'll get you line editing and history.The readline Wikipedia page mentions a library calledlinenoiseat the end, which claims to be much smaller (and BSD licensed), that might be worth a look for a microcontroller where size is an issue. On the other hand you mention some pretty advanced stuff in your wishlist, so that might not be right for you either.Another option is having only the simplest possible code in the microcontroller, and then run a shell around it on the controlling computer, if you have one; I sometimes use a small tool called "ile" (Input Line Editor) which provides editing and command history for dumb programs (like e.g. netcat).Hope that helps.

author: andyturk
author_rank: Frequent Contributor
timestamp: April 01, 2014, 06:39:15 pm
content: Not standard, but here's what I do to implement a command line shell. It's written in C++ (of course) and has ties to ChibiOS, but could be ported fairly easily to other environments.https://github.com/andyturk/libakt/blob/master/akt/shell.hhttps://github.com/andyturk/libakt/blob/master/akt/shell.ccSome sample output from a project using this via a USB CDC serial connection:Code:> ?exists -- test if a file is presenttype -- type file contentsrm -- remove a file or directorypwd -- print current directorymkdir -- create directoryls -- list filesdf -- show disk infocd -- change directoryi2c -- read/write to the I2C bussensor -- show sensor infocal -- show/clear sensor calibration info9250 -- Invensense MPU-9250 commandsreset -- force hard resetmemory -- display RAM/ROM infothreads -- list ChibiOS threadsinfo -- program/toolchain informationhelp -- list available commandstime -- show/set time of dayid -- show CPU unique iddate -- show/set calendar datebuttons -- show button statebat -- show battery charger state> dfFS: 237998 free clusters, 64 sectors per cluster, 7615936 kbytes freecard is writable> batexternal power is presentfullbattery voltage: 5246> i2c scanfound device at 0x60found device at 0x68> threadsaddr stku prio refs state time name20001AD0 0000 64 1 WTOREVT 00000032 main20001988 0000 1 1 READY 00047662 idle20003A18 0000 64 1 SLEEPING 00000000 button-debounce20003BD8 0000 64 1 WTOREVT 00000002 card-mounter20001F30 0000 2 1 READY 00000003 usb_lld_pump20002E58 0000 64 1 CURRENT 00000015 shell>

author: hamster_nz
author_rank: Super Contributor
timestamp: April 01, 2014, 06:47:58 pm
content: You could most likely hack my tiny basic port for Arduino to do something useful for you.http://hamsterworks.co.nz/mediawiki/index.php/Arduino_Basic

author: westfw
author_rank: Super Contributor
timestamp: April 02, 2014, 12:33:22 am
content: Thanks Magetoo! "readline" was what was on the tip of my tongue, but couldn't find ("Comfortable terminal input library" hmmph!) I don't like the license, though. I'll have to look into the alternatives that you and wikipedia mention.Thanks to others, too, but I was hoping for something well organized as a partitionable/extensible library, (don't need date/time parsing? Leave it out!) rather than examples of how to implement parsing in general. Rewriting CCMD might be reasonable, yet.

author: andyturk
author_rank: Frequent Contributor
timestamp: April 02, 2014, 06:04:58 am
content: A basic shell is not that hard to write yourself. The first task is to accumulate a line of text from a keyboard, UART, USB connection or whatever. Depending on how much line editing you need, it can be half-a-page of code (mine does backspace/delete and ^U to nuke the whole line). The trick to making it truly portable is is factoring the code so it'll work with blocking calls like getc() and also with interrupt routines that give you a character at a time.Then you'll need some kind of service entry point to see if a complete line of text has been collected, parse it, and look up the command to execute in some sort of list. Figuring out how to collect/represent the list of all commands known by the shell is the interesting part, IMO. Most implementations I've seen have commands appear in a monolithic table, which means that part of your code needs to know about *all* the commands supported by the system. That's a pain because when you want to add some new whizzy command for your latest project, you have to patch that in somehow to the list of boilerplate commands you want to be available in every shell. It's not conceptually difficult, it's just messy from a code management point of view.My approach was to skip the monolithic table and build up a linked list of available shell commands at runtime (via C++ constructors). I don't really care if looking up a shell command by name is O(N). The advantage is that simply constructing an instance of the shell command class adds it into the list, so the generic shell source code doesn't need to know about what commands are available. In fact, there's no list of commands in the source code at all--it's built at runtime.For instance, whenever you implement a new device driver, you can also add a shell command to talk to it. The shell command is closely associated with the device it's talking to and usually lives in the same source file as the driver. Add that driver to a new project, and you automatically get the shell command with it. Very cool.The first thing I do in a new project is blink an LED. The next thing is to bring up a command-line shell. It's a great way to get stuff running quickly.

author: amwales
author_rank: Regular Contributor
timestamp: April 06, 2014, 01:32:43 pm
content: If you are using C, you could take a look at lex, yacc or flex, bison, these are lexers and parsers.You normally start getting a little annoyed at the amount of code being generated and end up rolling your own lexer and parser.Alternatively, write up a simple grammar in BNF (http://en.wikipedia.org/wiki/Backus%E2%80%93Naur_Form),use a simple set of rules to translate that into code (http://en.wikipedia.org/wiki/Recursive_descent_parserlook at the C example).So long as you stick to the rules for translating it, coding it become a very mechanical process, it starts looking very verbose but is simple to write and extend later if you don'f fall into the trap of trying to optimize it.I hope that was helpful.

author: andyturk
author_rank: Frequent Contributor
timestamp: April 06, 2014, 04:16:49 pm
content: Quote from: amwales on April 06, 2014, 01:32:43 pmI love flex/bison, but they seem a little heavy for a command line shell, especially on a microcontroller. All you really need is to break up words by whitespace. Add quoted strings to that if you're going to be really fancy.

author: zapta
author_rank: Super Contributor
timestamp: April 06, 2014, 05:03:53 pm
content: Quote from: andyturk on April 06, 2014, 04:16:49 pm+1BTW, I liked your idea of independent commands register themselves using static constructors and having a common base class.

author: amwales
author_rank: Regular Contributor
timestamp: April 06, 2014, 07:04:56 pm
content: Quote from: andyturk on April 06, 2014, 04:16:49 pmThis is why I said in my post - "You normally start getting a little annoyed at the amount of code being generated and end up rolling your own lexer and parser."

author: legacy
author_rank: Super Contributor
timestamp: April 06, 2014, 07:58:19 pm
content: Hiinthisthread i presented my little interpreter written in c++, sources is provided following the git repo.feel free to modify it as you want

author: Erwin Ried
author_rank: Regular Contributor
timestamp: April 07, 2014, 05:02:14 pm
content: Have you checked "bitlash"?http://bitlash.net/

author: codeboy2k
author_rank: Super Contributor
timestamp: April 08, 2014, 07:46:57 am
content: Quote from: andyturk on April 02, 2014, 06:04:58 amI agree with the turkGetting an interactive monitor/shell running is a best practice first step.I sometimes have done this with a forth shell. The basic primitives are there in the default interpreter, lots of libraries are reusable, and I can create device drivers interactively in forth to test out stuff immediately. It's really great for embedded work and a seriously lost art.

author: Rigby
author_rank: Super Contributor
timestamp: April 08, 2014, 01:21:51 pm
content: I don't know a lot about microcontrollers or microcontroller programming, yet. When I write a 'shell' in C# or Java, though I use what is called the "command pattern" and if you're using an OO language on your micro, such as C++, then the pattern would work there, as well. The command pattern can be used to do parse just about anything that comes in a string, even weird things like Roman numerals.

author: linux-works
author_rank: Super Contributor
timestamp: April 08, 2014, 06:35:55 pm
content: the high level things in c++ and c# don't really belong in controllers. there is VERY limited memory there and even limited code space in the arduino, for example. using OO things starts to bring in too many libs and dependancies. something is classed off of something else and so they all get pulled in. blech! not good for low mem controllers.(OP: is this bill from the old cisco days?)

author: Rigby
author_rank: Super Contributor
timestamp: April 08, 2014, 07:24:26 pm
content: I have a micro on my desk with 1MB of program space. I'm sure I could fit a few objects in there. I'm not talking about C# at all beyond it being an OO language - it doesn't really suit my needs as a development platform for microcontrollers.You're definitely right for most micros, though. There's probably a way to do it without objects, though, and in quite small space, which I guess is what OP is asking about.

author: linux-works
author_rank: Super Contributor
timestamp: April 08, 2014, 07:51:27 pm
content: the arduino 328 chip (most popular of the arduinos) has 32k (k-bytes, not m-bytes!) of program space!!that amounts to about 10k lines of C code. that's all. I filled my arduino up almost to the last byte with about 13k lines of c code on a project I was on. I had to remove some things in order to add others; I kept running out of the 30k (2k goes to bootloader) that the arduino allowed. I even had to remove floating point calls (libs) since that ate up 4k or so of code. 4K!! not meg.c++ is totally wrong for small footprint embedded use. you can pull all kinds of things in by mistake and not even realize it.

author: Rigby
author_rank: Super Contributor
timestamp: April 08, 2014, 09:42:42 pm
content: Yeah, I know. I'm agreeing with you.My micro with 1MB of program space is theTI Tiva-C Connected Launchpad. It's actually a pretty great board, and only $20 shipped.

author: linux-works
author_rank: Super Contributor
timestamp: April 08, 2014, 10:10:31 pm
content: ah, to have a full megabyte.if only to dream...my code was so full of ifdef's since if you wanted this feature, it would be at the expense of another and you'd have to be clever with how you ifdef'd things. there simply was not enough code space in the controller and so it had to be structured as optional modules that you would include only at build time, certainly not at runtime.a lot of libraries and systems for unix don't take into account low mem footprint size, for target binaries. you have to pick very low dependancy libs if you want them to fit in tiny controllers.when I have to parse strings inside controllers, I just use the string.h primitives and go from there. its more work, but I have no lib dependancies other than libc, which has to be there anyway.

author: Rigby
author_rank: Super Contributor
timestamp: April 08, 2014, 10:37:15 pm
content: sounds like a sound approach to me. aren't there microcontroller libs available now that have been honed a bit to produce smaller binary sizes? github probably has a bunch of microcontroller gems littered about in random places.

author: senso
author_rank: Frequent Contributor
timestamp: April 08, 2014, 11:11:56 pm
content: The arduino IDE produces a lot of crap, blink a led 1.x k, do it with avr-studio/c+makefiles, and its 184 bytes, a little diffference.I'm also dipping my toes in this subject, so keep posting about this, so I can get some ideas, with luck it will work at the 100 try

author: westfw
author_rank: Super Contributor
timestamp: April 09, 2014, 01:53:47 am
content: QuoteYep. Do I know you? It's ... annoying to have all this experience with a proprietary infrastructure that I no longer have access to :-(One doesn't usually do much of a CLI parser on small microcontrollers, but now that they're getting up to the 256k to 1M code size, it's worth thinking about again. Although the other annoying differences a modern microcontroller with 1M of memory and a original PC with 640k of memory is that the PC memory was all RAM, and people wrote code without paying any attention to which parts of their data could/should be read-only. (Slightly interesting: the original tops20 comnd% jsys was better than the CMU CCMD libraries, out of desire to keep code pages shareable...)

author: mariush
author_rank: Super Contributor
timestamp: April 09, 2014, 02:07:00 am
content: If you're low on actual space in the flash, you could always store the commands in the eeprom - 1024 bytes on the ATmega328, 512 bytes on the ATmega168 and ATmega8, 4 KB (4096 bytes) on the ATmega1280 and ATmega2560you can store it for example like [1 byte for command length] [ x bytes for command 0] , [1 byte for command length] [ x bytes for command 1] ... repeat [ null to tell you end of commands]then you have a switch or a bunch of ifs in the code.when reading from eeprom, if the command you parsed has less or more characters than the value you read from eeprom, you can just skip to next command in eeprom

author: westfw
author_rank: Super Contributor
timestamp: April 09, 2014, 02:42:37 am
content: One of my early attempts at fame was a Serial Comms program for the IBMPC. It had to run in 256k of RAM, and from a single 360k floppy. Lots of 8086 assembler. Unlike a lot of SW from that timeframe, it had extensive help for its (also extensive) configuration menus. But the help text was all stored on the floppy till needed. That actually worked pretty well (relative to the standard of the day...)(That never did see the light of day. What an education in business details and sleazy-acting partners. And lawyers. Oh well...) (and looking back, it had some really cool security holes (like an H19 escape sequence that displayed internal memory values...))

author: mrflibble
author_rank: Super Contributor
timestamp: April 12, 2014, 12:04:47 pm
content: Quote from: westfw on April 09, 2014, 02:42:37 amIBMPC, are you still there? If so, reply "HAT" (500 letters).

author: mrflibble
author_rank: Super Contributor
timestamp: April 12, 2014, 12:11:52 pm
content: Quote from: codeboy2k on April 08, 2014, 07:46:57 am*grin* Same thing here. First up is the mandatory led blinkie. Cannot have a good project if it didn't start life with a led blinkie.And yes, a shell is quite helpful. Also, I find it makes the development process more fun. In the sense that coding up some new functionality, and then getting immediate output in response to test commands can be very rewarding.Yes you can stare at variables in the debugger but it's not quite the same as typing a test command and then getting"Thingy::Stuff() here: your answer of the day is 42."on the serial port.QuoteThe "create device drivers interactively" part sounds intriguing. In broad terms, how would that work?

author: SL4P
author_rank: Super Contributor
timestamp: April 12, 2014, 12:20:36 pm
content: Quote from: linux-works on April 08, 2014, 06:35:55 pmI'll see if I can find it... a long time ago I wrote a symmetric 'Cisco-IOS-like' command parser - that would allow you to dump the config via the terminal in ASCII, and reload it directly into the command line (using a terminal app). Very useful, along with a human readable copy of the config settings.

author: andyturk
author_rank: Frequent Contributor
timestamp: April 12, 2014, 03:43:33 pm
content: Quote from: mrflibble on April 12, 2014, 12:11:52 pmDon't know about using Forth, but I use a generic "i2c" shell command that talks to anything on the bus:Code:> i2c ?i2c -- read/write to the I2C bussyntax: i2c r <addr> <reg> <length>i2c w <addr> <reg> <bytes>i2c scan<addr> and <reg> are 2-digit hex numbers<bytes> is an even-length hex byte string<length> a decimal number greater than 0> i2c scanfound device at 0x60found device at 0x68> i2c r 0x68 0x00 16read addr=0x68, register=0x00, len=16c4 cd e0 e9 a0 00 16 2f ee 1a fb fd 00 5f 59 7c>When developing a driver for an I2C device, you're mostly just sending and receiving short byte sequences. Being able to figure those sequences out without having a compile-link-flash in between each attempt is pretty handy.

author: legacy
author_rank: Super Contributor
timestamp: April 12, 2014, 08:47:44 pm
content: about lex & yacc, who is good at them ?i am learning them, i downloaded C99 grammar, i arranged lex and yacc sources and … i am having issues with "typedef".i meantypedef { blablabla } my_t;that is finebutmy_t foo() {}is not fine cause i am not able to say "hey, my_t is a new type"so my problem is that i have pre-defined types. I am stopped at this pointlet me know if you need the full sources

author: westfw
author_rank: Super Contributor
timestamp: April 12, 2014, 11:36:12 pm
content: Quotepresumably you have to define the my_t portion of your function definition as a generic identifier, and check later whether it's a valid type name, or you have to have the list of typenames that you accept in a function be dynamic somehow.IIRC, C does not have a "pleasant" grammer when it comes to parsing with regular tools. (this is one of the reasons that C compiler error messages suck so badly.) (Everyone liked pascal, though.)

author: miguelvp
author_rank: Super Contributor
timestamp: April 13, 2014, 12:33:03 am
content: I know this is overkill for what you want, but just throwing it out there.There is a hardware xml parser out there and I've seen fpga based parsers that can process input very fast. But yeah, that's for something more than just reading a command line, but if you need speed for reading those scripted inputs or if they are networked commands or many other applications, then maybe put in some hardware to do it.

author: codeboy2k
author_rank: Super Contributor
timestamp: April 13, 2014, 01:14:47 am
content: Quote from: andyturk on April 12, 2014, 03:43:33 pmsimilar with Forth, except the basic forth shell is a small interpreter written in C that has the basic primitives (called words) for reading and writing memory or reading and writing I/O space.So if the shell is loaded I can write a new word called 'i2Creadbyte' which reads a single byte from a device on the i2c busthen I can build up on that word, to write a new forth word that can read multiple bytes, i2creaddataSimilarly I can create new forth words for i2cwrite and i2cwritedata.Then I can build up on that word further to write a new word that be a device driver , using the previous i2creaddata and i2cwritedata.If the i2c device is a 2x16 line display, for example, I can then write a new word "printmsg" that uses the new i2cwritedata words.basically, the forth shell is interpreted, interactive and can create new primitives (words) that can also be used inside more new forth words.The forth shell is a toolset of primitive words and more complex words built up from those primitives.From Wikipedia:Quote

author: andyturk
author_rank: Frequent Contributor
timestamp: April 13, 2014, 01:40:35 am
content: Quote from: legacy on April 12, 2014, 08:47:44 pmFirst off, this is probably not a good place to get help/advice on parser generators. Second point, why aren't you using flex and bison? Third, there's more to parsing than just copying someone else's grammar. You're also going to need a symbol table to keep track of things like 'my_t'.You'll learn a lot more about parsers if you start writing your own simple grammar. Fixing reduce/reduce errors on a big grammar is no fun. In fact, if you're just starting out, try implementing a recursive descent parser because it's easier. You can use lex/flex for the tokenizer, but recursive descent can be done just with carefully written C functions--no need to generate code.

author: legacy
author_rank: Super Contributor
timestamp: April 13, 2014, 02:24:41 pm
content: Quote from: andyturk on April 13, 2014, 01:40:35 amshould i open a threadQuote from: andyturk on April 13, 2014, 01:40:35 ami am using flex, lex, bison, yacc, all the stuff you can get from linux (i can emerge whatever you suggest me, i am on a gentoo machine), it is my first time with these toolsQuote from: andyturk on April 13, 2014, 01:40:35 amyes, but i have read a fewlex & yacc tutorials, which are very easy about grammar, in fact the example is a "calculator", which i am able to write, and i wrote, by my hands.Quote from: andyturk on April 13, 2014, 01:40:35 amI have also written a Pascal-similar interpreter, you can find it in the link i have put in the first page of this thread. It is written easily, without any tools.hereit is the link of my thread, it is written in C++Quote from: andyturk on April 13, 2014, 01:40:35 amjust asking for a tips&tricks, i repeat, it is my first time with yacc and C, also i need a C grammar implemented in order to realize a sort of sub-MISRA code validator. Too much work about designing it by hands, i mean, for me.

author: legacy
author_rank: Super Contributor
timestamp: April 13, 2014, 02:54:01 pm
content: Code:# obj/my --debug[ .k.a.n.o.j.o. ][pico.slu.sh.ell]v4.2m, 2013compile info:branch 4th_issue_r0.2/2013# helpkanojo: in Japanese to refer to the female 3rd personhelp................ - hope it can helpcmdlist............. - list of all available cmdscmds................ - list of all available cmds?................... - list of all available cmdsver................. - show the versionenv................. - show all the environment entriesenv_show............ - show all the environment entriesenv_save............ - save all the environmentenv_set............. - edit/create an environment entryenv_cp.............. - cp env into new envexit................ - force an exitrpn................. - rpn a linea (experimental, it points to "is_expr_valid", no action perfomed)# rpn "var1+var2+(var3*var4)"parsing ("rpn" var1+var2+(var3*var4)""), argv[]={ [var1+var2+(var3*var4)] }invoking lexer (from lib_lexer)sentence=[var1+var2+(var3*var4)]1: ok2: 3: tokens_show v[]={ 0 1 2 3 4 5 6 7 8 } ntoken=9- token: token= TokenAlphaNum- token: token=[+] TokenPlus- token: token= TokenAlphaNum- token: token=[+] TokenPlus- token: token=[(] TokenOpenBracket- token: token= TokenAlphaNum- token: token=[*] TokenAsterisk- token: token= TokenAlphaNum- token: token=[)] TokenCloseBracketinvoking rpn_shunting_yard with [var1+var2+(var3*var4)]input: var1+var2+(var3*var4)-------- to_lexer begin ---------------- rpn_core_for_lex begin ----------------- rpn_core_for_lex end ---------output v[]={ 0 2 1 5 7 6 3 }{token= TokenAlphaNumtoken= TokenAlphaNumtoken=[+] TokenPlustoken= TokenAlphaNumtoken= TokenAlphaNumtoken=[*] TokenAsterisktoken=[+] TokenPlus} stack_balance=1-------- to_lexer end --------########################## good expression ##########################ans=(1)I have also written this toy-tini-shell with an RPN insideit has its lexer libraryall manually designed and written by my handsbut … yacc seems more difficult for me ='(the idea about this project aims to the same target of this thread!