Pass commands to an MCU to execute, is there a standardized format? - Page 1
Forum section: electronics / general

author: jnz
author_rank: Frequent Contributor
timestamp: December 29, 2016, 06:54:49 am
content: I have an ARM MCU project that will store a list of commands from a remote host packaged into a simple binary format and run through a parser/executor style program. Nothing too complex. These are not MCU commands but higher level things the app knows how to do.Is there a standardized format for how lists like this should work? Maybe how lists like this are packed with cmd/meta/data/len/crc-repeat-repeat? Any sort of "well, if I had to write an interpreter to read in a list of variable length commands I'd probably make it look like XYZ", etc.I know a lot of this will be application specific, but if there are some best practices or examples of packaged and parsed commands I can look up that would be great. Thanks if you can help!

author: Ian.M
author_rank: Super Contributor
timestamp: December 29, 2016, 07:24:01 am
content: There's more ways of implementing abytecodeinterpreter than you can shake a stick at. Also you have to decide if a simple sequential scripting engine, possibly with bounded looping is sufficient or whether you need it to be Turing complete. If so, it will reduce your workload considerably if you implement an existing language that tools are available for.Also, who's going to write the scripts? If its going to be end users, you may wish to consider a BASIC interpreter. If its going to be general developers Java Card for ARM may meet your needs. If its going to be only your own team, and you need a very lightweight implementation, consider using a FORTH interpreter. There's a fair number of them written in C.

author: jnz
author_rank: Frequent Contributor
timestamp: December 29, 2016, 07:32:17 am
content: Quote from: Ian.M on December 29, 2016, 07:24:01 amyes.QuoteJust me and my team. For I'll look into fourth for some ideas. Basically the whole thing revolves around having 20 commands that need to be in different orders with different data so I'll be databasing these lists and just serving up the correct one to the device when needed.For problem customers it would be nice to custom generate a job/list of commands for them that solves their specific issue. Something I could only support with a firmware update now.

author: voltsandjolts
author_rank: Unknown
timestamp: December 29, 2016, 08:34:24 am
content: There are existing interpreters for microcontrollers which implement eg. BASIC, scheme, forth, javascript, spin or Python (micropython). Maybe more complex than you need but they are done already.Maybe you need something obfuscated?

author: jnz
author_rank: Frequent Contributor
timestamp: December 29, 2016, 05:47:46 pm
content: Quote from: voltsandjolts on December 29, 2016, 08:34:24 amNot a big fan of obfuscation. The commands I'm sending from remote to device are encrypted for transit already.It's more than I don't want to come up with my own byte level packing system and interpriter if I don't have to - or at least could learn some best practices. There just isn't enough room in firmware to store all the variations of code that needs executing in a specific order, so I need to store the right ones in a database and execute remotely, I'm 90% sure it's going to be my own implementation, but I figured I'd ask!

author: rstofer
author_rank: Super Contributor
timestamp: December 29, 2016, 06:37:30 pm
content: Build up a command language using BNF (like the definition of most computer languages) and decide how far you want to take things. A similar but more intuitive approach is to design the language using syntax diagrams.I Googled for "computer noun verb object" but "subject verb object" might work as well.https://books.google.com/books?id=Xb3sguO2QjQC&pg=PA34&lpg=PA34It turns out that Pascal is defined with syntax diagrams and the recognizer code flows exactly from the diagram. There is a tiny Pascal compiler and run time system in the back of "Algorithms Plus Data Structures = Programs" by Niklaus Wirth.See syntax diagrams for full Pascal here (page 47)http://www.standardpascal.org/The_Programming_Language_Pascal_1973.pdfWhen you look at the compiler code, every block of code flows exactly like the diagram. In the source, the main function calls 'block' and, upon returns, looks for a period '.'. See the definition of 'program' at the bottom of page 49.Pascal has sets as a fundamental data type. That's nice because the recognizer uses two sets of symbols: Those that can start a statement (used loosely) and those that must follow (like a semicolon). The elements of the begsys set change as the recognizer wanders through the syntax diagram.Anyway, the point is this: develop a simple language that will produce all possible commands, make the grammar unambiguous, and build a recognizer right from the syntax diagrams. Nice and clean! More important, the recognizer will reject errors by skipping symbols until it finds a matching symbol in endsys set (usually the ';' but also the '.'). In other words, have a command terminator even if it is just a CR. You could also allow multi-line commands by realizing you are inside a 'block' command like Python does in Idle (command interpreter).

author: Ian.M
author_rank: Super Contributor
timestamp: December 29, 2016, 06:51:36 pm
content: If you base it on Pascal, you've still got to write the embedded P-code (bytecode) machine . As the bulk of the firmware is probably written in C, the availability of a bytecode machine written in C that will be easy to integrate with the rest of the firmware will be a strong factor in choice of scripting language.Caution: if you choose to implement a Turing complete bytecode machine and any of the primitives can access arbitrary non-sandboxed memory, your system is insecure by design.

author: jnz
author_rank: Frequent Contributor
timestamp: December 29, 2016, 07:16:03 pm
content: Nope.See, I made this WAY TOO complex.I get this attached. The idea about making is PASCAL or PASCAL-like. I get the warning from IAN about allowing an arbitrary to access non-sandboxed memory... And for the question I asked you guys are all right. The issue is the code is not accessing MCU functions and memory. It's accessing MY functions. And only my functions. If I had the room or knowledge about what a customer needed their device to do before hand I'd be happy to hardcode all of the function in but that's not happening.So I have functions in the app like this:Init_Comm(ARG1);Read(ADDRESS);Write(ADDRESS, LEN, DATA);Reset();They're complicated in their own right. But they exist and work. The issue is that Customer X and Y are hooked devices that need different order of operations.Customer X needs:Init_Comm(NULL);Read(0x0123);Write(0x0123, 1, 10);Reset();Customer Y with their device needs:Init_Comm(SPECIAL_TYPE);Read(0x0111);Write(0x0000, 1, 10);Write(0x0111, 9, 001122334455667788);Reset();Now, push that out to dozens and dozens of unique devices and can't store on the device for various reasons. The the most practical let's say the remote device knows which set of instructions works and needs to push that to the customer's device.My idea.... obviously... was to assign a simple format to the know instructions I have and pack the data in a known size. However, this "API" is entirely arbitrary. So something like0x11 INIT + 1 argument byte0x22 READ + 2 address bytes0x33 WRITE + 2 address bytes + 2 length bytes + the number of bytes in length0x44 RESET + 0 additional byes.CustomerX's example above packed might look like:11-00-22-01-23-33-01-23-01-10-44, with a total_num_bytes and a CRC tossed in there.But... Maybe there is a better way to pack that? Maybe there is a good argument for starting each command with it's own total length up front on the first byte? Maybe there are some standards to follow or examples to learn from other projects?I like the idea of the attached looking for , and ; symbols, however, bandwidth for these command lists is already at premium and I already know how many bytes I'll be expecting or can easily check. So I should be able to determine where this line item command ends the next command begins.Is that anymore clear or did I make things worse?!

author: pitagoras
author_rank: Regular Contributor
timestamp: December 29, 2016, 07:31:14 pm
content: From the main application point of view, it would probably like sending HTTP/REST with JSON data structures.for examplePOST /controller/ADDRESSwhere ADDRESS could be for example 0x883and sending a json object in the post such as:{"data": "9231klj3lkj1291j32jl2", "otherThigs": 223}but as this probably is too cumbersome in the controller side (i.e. needing an http server) you could just simplify and make a REST like protocol. At least you have the design framework and semantics. So the commands would be<verb> <resource> <jsonData>where:- verb is GET, POST, PUT, DELETE (and there are others). Get is to obtain data, POST is to send data, PUT is to send data idempotently, DELETE is... well "clear"- resource is a "path" you define for your resources. For example /port/0x222/counter- jsonData is a json object as described earlier (used in put)For exampleGET /port/0x228would return{data: "2jk1jh312kh3jkh1jkh3jkh2jk3"}PUT /port/0x228 {data: "asd239842j34j234u3"}In REST you also have the http defined return type404 "resource not found"200 "operation ok"500 "internal server error"that you can map conveniently in the response, i.e. a certain GET can return 200:{data: "2jk1jh312kh3jkh1jkh3jkh2jk3"} if everythin OK, or 404 if the port does not exist, of 500 if there is some problem in the controller.App side developers should be happy

author: Ian.M
author_rank: Super Contributor
timestamp: December 29, 2016, 07:41:43 pm
content: So you are looking for something low level, dead simple and not Turing complete. As you already seem to be heading that way I agree that you should consider using single byte command codes. Each command knows how many parameters it uses, and their types: either fixed length, or as a length + a block of data bytes, so knows how many bytes to consume before passing control back to the interpreter command loop. Write functions for each type to consume and return the next parameter. If you need more than 255 commands, use 'expansion' codes that switch to a different command set for the next command only. Put all the most frequently used commands in the base command set.If you have to support looping, jumps or conditionals it gets a little more complex and you may need to build a table of command lengths and offsets when the script is downloaded so you can branch N commands forward or back or goto the Mth command (much safer than arbitrary relative jumps to an offset in the command byte stream), and also, for robustness, may want to include length bytes so corrupted commands that could cause data to be executed as commands can be detected and the whole script rejected.

author: jnz
author_rank: Frequent Contributor
timestamp: December 29, 2016, 07:45:09 pm
content: Quote from: pitagoras on December 29, 2016, 07:31:14 pmI like that. Making it function like JSON objects, and it would pack fairly well, I could even take ques from JSON packing although once it gets to me it's really arbitrary with how I treat it. That said...That treating it like JSON looks really good for individual commands. Replying with 200 on a success for example. But I don't have the latency or bandwidth to run individual commands and report and answer to anyone, I pretty much have to download a list of commands and execute all of them. Still, no reason I couldn't model it after JSON as that's pretty similar to the idea.QuoteHa!

author: jnz
author_rank: Frequent Contributor
timestamp: December 29, 2016, 07:49:27 pm
content: Quote from: Ian.M on December 29, 2016, 07:41:43 pmOk, so that's pretty much what I laid out and what I thought. For all I knew this was a common CS thing that had a bunch of well established practices, but it seems pretty much do whatever you feel like.I had the same idea on having each function know it's own length of bytes needed, I can switch for the 0-255 command and in the next few lines eat up the next bytes as that function dictates. If I combine that with some of the JSON-like stuff mentioned above I think I have almost everything I'll need.Thanks for the thoughts on this.

author: Ian.M
author_rank: Super Contributor
timestamp: December 29, 2016, 07:59:55 pm
content: The other thing to consider is how easy will it be to build the bytestream from a human readable scripting language using your toolchain? You may be able to get away with abusing the C preprocessor to generate a const array of char, with a #defined macro for each script command, but that leads to - at best - cryptic error messages and much pain debugging it. If you write a script compiler/decompiler, its more effort up front but the scripts will be far more maintainable. You'll also need a bin file to C array utility so you can do a single MAKE build of your complete firmware with its default scripts.

author: tggzzz
author_rank: Super Contributor
timestamp: December 29, 2016, 08:30:10 pm
content: Let me guess the history and motivations, and predict the future.You have some form of control device, and you haven't fixed everything itneedsto do. So the idea is that you will retain flexibility by having some form of very simple scripts that allow you to change behaviour after development is "finished".Let me predict how this will pan out, based on my having been in this position once and observing several others over the decades.You will invent a domain specific language to satisfy today's needs. Next month new needs will be found, and the DSL will be extended a tiny bit. And then another tiny bit. And then another tiny bit...After a while the scripts will grow and grow and grow - and the DSL features will grow and grow and grow until they all start interacting in a foul way, and nobody can really predict how things will work. Inventing tooling will be too costly, and the designers will want to move onto new great things, so they hire a developer to take over their work. The only people that would take on such a task are naive losers that don't read WTFdaily.The alternative is to create primitive operations, and to use a standard language (with tooling and knowledgeable engineers) to invoke the primitives, optionally via your own domain specificlibraries. If the language runtime contains a REPL interpreter, then the scripts become simple ASCII scripts.Which approach do you think is easier in the short term and long term? Hint: both answers are the same.(There are several suitable languages, depending on your constraints. Forth, JavaScript, Scheme, and Python spring to mind)

author: diyaudio
author_rank: Frequent Contributor
timestamp: December 29, 2016, 08:39:20 pm
content: Quote from: jnz on December 29, 2016, 06:54:49 ama few bytes, array command lookup table. done.The beauty of simplicity.

author: jnz
author_rank: Frequent Contributor
timestamp: December 29, 2016, 09:42:23 pm
content: Quote from: tggzzz on December 29, 2016, 08:30:10 pmWhile I do appreciate the advice and can see where you're coming from, a lot of the concerns you talk about are null due to the fact that as per my example a READ command is already defined in the target system. In fact, all of the commands are defined in my target system, there is nothing more I can do or add that isn't there already as I am interfacing to this other box with a well standardized and defined system in place. So the domain specific language really has no where to grow to, but I will take the advice and consider it.Quote from: diyaudio on December 29, 2016, 08:39:20 pmPretty much, but there are some options due to the entirely arbitrary nature of it. This forum is really good for people going over my head quick with complex options, but it helps to take the advice and scale it back to something I can wrap my head around properly.I have maybe 20 commands and they each have an argument or two of known size or a length I can read to determine that command's whole size. I'll be fine with simple I think.

author: tggzzz
author_rank: Super Contributor
timestamp: December 29, 2016, 10:16:42 pm
content: Quote from: jnz on December 29, 2016, 09:42:23 pmIt sounds like the READ is a primitive. Beware of wanting to store and process the value read. And then make a decision on its value. And then repeatedly READ until a particular value is found. And then doing it in two places in the script.Better to define, say, a Forth primitive that invokes the READ.

author: Buriedcode
author_rank: Super Contributor
timestamp: December 29, 2016, 11:07:43 pm
content: Whilst I'm hardly experienced in this, it does seem like there are two camps in this thread - those from a high level background, of very complex systems (leading to almost creating ones own scripting language) and those who just see a microcontroller with a handful of functions that need to be implemented upon receiving a command (with associated data). From the OP's posts, it seems like he's doing the latter, and of course there should be the facility to add more 'commands' perhaps even processing the data differently depending on how many bytes there are following the command - so it can easily get messy.But I really don't think the OP is after 'scripting'. Rather the sort of thing we've all done which is a very basic format (that may be too limiting after a few versions) such as <command><bytecount><data><checksum>. In which case, I haven't found any 'popular standards' although if its part of a network then there are of course addressing concerns. I don't have my own standard and 'play it by ear' which I dont' think is best practice - being called to update a system I designed years ago means having to dig out the notes and firmware and reacquainting myself.As suggested, the simplest would be a look-up table of a list of functions. Command data would need to be passed to the function, or, heaven forbid, put in a static array, but for a few bytes of data, and for quite a few commands, it'll do the job. I often used two ASCII characters for the command to make it semi-human-readable, ASCII for entering values, pretty much so a terminal and a list of commands is all that is required for testing. Also chose fixed length packets for simplicity and when it wasn't required to run hundreds of commands a second.Admittedly, this was all for 8/16-bit AVR/PIC applications. Of course I don't know what the application is, so I have no idea of complexity and I'm assuming its simple..

author: tggzzz
author_rank: Super Contributor
timestamp: December 29, 2016, 11:18:34 pm
content: The <command> etc you mentionisa DSL, albeit a very simple domain specific language. I'verepeatedlyseen that concept mutate into a cancerous mess.Often the solution can be implemented in the form of a domain specific language or a domain specific library. One builds on the work of many other people, the other reinvents the wheel (and that wheel is usually eccentric).The scripting and the DSL are orthogonal design decisions, but the two are often combined.

author: rstofer
author_rank: Super Contributor
timestamp: December 30, 2016, 12:43:44 am
content: Your 'sentence' recognizer can exist in one of two places: On the device or on the client machine. It matters not at all which system does the recognizer task.If the recognizer is going to be on the client machine, your device can 'assume' that the incoming command stream is valid. After all, it was sent by a perfect recognizer. It might look for some trivial failings but, really, the incoming command stream should be 'perfect'.On the client's machine the recognizer will look at incoming sentences and compile them into some 'perfect' intermediate form.This all falls in a hole if you can't install an application on the client machine to deal with compiling sentences.I don't think the Pascal is a necessary language just that I really like syntax diagrams. Among other things, it makes obvious to users exactly what sentences can look like. Probably good documentation.

author: pitagoras
author_rank: Regular Contributor
timestamp: December 30, 2016, 12:59:12 am
content: What about just deploying a native code to the MCU and doing the translation in the cliente app.At first you can just have snippets in the client application that map "commands" to native code fragments.For exampleRead(0x333)maps to (sorry just some random assembly language follows, I don't know your specific assembly code)LDA 0x33STA 0x33Note this code could be generated with a proper compiler in the same build workflow as the client application.Then in you client app you can do some typical DSL builder likeMCU.initCom(null).read(0x33).write(0x222,1,0).reset().buildCode()which builds a piece of MCU code to be deployed over wire and executed. You just need some loader in the MCU side

author: Jeroen3
author_rank: Super Contributor
timestamp: December 30, 2016, 07:00:30 am
content: How about actually using compiled code? You can make the chip run actual code from SRAM, why not put some actual code in there calling functions you've put in flash?Compile your "interpreter" as a library, to which you can compile your "script". Which you then load in sram, and execute. Do you need features not available in your "interpreter", simply add it to your "script", since it has the same capabilities as any code.Basically just how you'd use a ROM library, as in some NXP parts.

author: voltsandjolts
author_rank: Unknown
timestamp: December 30, 2016, 08:52:13 am
content: Or use a microcontoller that has no flash at all and gets it firmware from USB every boot and runs from RAM. Just load different firmware for each task you have, maximum flexibility and an existing solution.So many options!

author: donotdespisethesnake
author_rank: Super Contributor
timestamp: December 30, 2016, 08:54:35 am
content: Well, this thread is a great example of how people love to over-engineer things! There is no need here for an embedded script language.For the OP, there is no "standard" way to tokenize arbitrary text, why would there be? Tokenizers for specific languages are specific to that language.Sticking to the KISS principle, it's very simple to define you own byte code, you are already on that path. You know the range and syntax of your tokens, it's just a question of finding an efficient packing.

author: voltsandjolts
author_rank: Unknown
timestamp: December 30, 2016, 09:06:09 am
content: QuoteSticking to the KISS principle, you could use something that is already done and tested, like an embedded scripting language.

author: tggzzz
author_rank: Super Contributor
timestamp: December 30, 2016, 09:41:10 am
content: Quote from: voltsandjolts on December 30, 2016, 09:06:09 amMany people with an academic computing background long to create their own language and write their own compiler. The electronic engineering equivalent is creating your own microprocessor.Those are fun activities with some merit, but in almost all cases the productivity (as defined by skinning a cat) is, um, suboptimal.