Configuration file format - Page 1
Forum section: electronics / projects

author: css
author_rank: Newbie
timestamp: March 18, 2018, 12:09:23 pm
content: Hi allI am developing a distributed embedded system and need your help on a decision regarding device configuration.First a little background:I develop network nodes (based on 32-bit MCU programmed in C). These nodes all carry a configuration file defining their behavior. The nodes can be firmware upgraded, which may change the structure of the configuration file. The nodes in the system can have different firmware versions and as a result, differently structured configuration files. The nodes are not always connected to the server (maybe 10% of the time – the remaining, they are “out of reach”).The nodes all connect to a central server to deliver data. Also, the server should be able to change the configuration of the nodes. When the nodes connect to the server (remember they are not always connected), they upload their current configuration.The server maintains a list of nodes connected to the network and the latest configuration file received from each node. The interface to the server is a http web service, listing all the known nodes and enables changes to their configuration files.Now the question:To simplify above compatibility between the server and the nodes (with different firmware versions), I seek a new configuration file format. Ideally, the format should satisfy the following:- Easy to parse on a small embedded device- Easy to parse on the server- Easy to present to the user with the web service (on a webpage)- Be self-explanatory (such that the server does not need to know thestructure of all configuration files) Contain valid ranges, datatypes etc. Such that the server can limit the user to select onlyvalid parameter valuesI hope to find a “standard” format similar json, ymal, xml which includes the “schema” (such that the format and valid parameter values are contained in the format), which can easily (preferably using existing open-source code) be visualized and edited on the webserver using html forms.It seems to me that many of the often used configuration file formats lack the ability to be truely self-explanatory" as the "schema" is stored in a separate file.I hope above is clear and appreciate any assistance you can provide.Best regardsChristian

author: rhb
author_rank: Super Contributor
timestamp: March 18, 2018, 02:27:55 pm
content: I've written parsers with lex and yacc, though quite some time ago on my first job as a contractor with a major oil company. My first assignment was similar to your problem. I had just spent 6 weeks learning lex and yacc. I wrote a bulletproof parser in a couple of weeks while spending about half my time doing other things. They had allocated 2 months for doing the parser.I suggest some variant of:value:name:type:size:min:max:default:commentas plain ASCII text. It's easy to parse and fairly obvious to a user and the server doesn't have to care whether the client uses all the information. The major advantage is that there are no order dependencies. Unless you already know lex and yacc, I don't suggest you use them. If you did, you probably would not have asked the question. Simply write a parser from scratch in C. Just watch out for side effects in strtok(3c). It is *not* reentrant.

author: ejeffrey
author_rank: Super Contributor
timestamp: March 18, 2018, 07:23:02 pm
content: INI files are semi-standard, have plenty of open source parsers, and are fairly easy to handle on a microcontroller. The main problem is that interpretation of the values is unspecified -- there is no standard to denote a numeric vs. string type, much less compound data types.A fully functional parser for most of your generic file formats (XML, JSON, YAML) is going to be fairly large for most embedded applications. One alternative you can consider is to use an extremely restricted subset of one of these standards, such that you can easily write an ad-hoc parser on the micro side. For instance, if you just need scalar values, you can require that a configuration file is just a single mapping of keyname -> value, where every value must be a string, number, or boolean. By eliminating list and mapping types, it is much easier to write a parser for JSON or YAML in a restricted footprint. The server can enforce the necessary restrictions, as well as enforce a schema of the legal keys and their data types. This would also leave room to grow in a backwards compatible fashion if you need greater flexibility. If your MCU supports it (or can be upgraded), you just drop in a fully featured YAML parser and you get all the functionality of that.

author: Dubbie
author_rank: Unknown
timestamp: March 18, 2018, 07:26:29 pm
content: I have had a good experience using JSON and the jsmn parser. I used to use ini, but much prefer JSON these days.https://zserge.com/jsmn.html

author: sleemanj
author_rank: Super Contributor
timestamp: March 18, 2018, 07:37:45 pm
content: I'd use json probably, especially since web is in the mix. I expect you can find a full parser library fairly easily, or a simple recursive decent parser for a subset should be fairly trivial. I would not use xml, it's too complex for what is needed in most configuration data, and probably wouldn't use ini since it's not really a well defined standard.

author: David Chamberlain
author_rank: Frequent Contributor
timestamp: March 18, 2018, 09:33:40 pm
content: I also would go with JSON. You could include the schema in the same file like this.Code:{"schema":{"user":{"type":"object"},"user_name":{"type": "string","length_min": 5,"length_max": 50,"filter": "/[^a-zA-Z0-9]+/"}},"data":{"user": {"user_name":"bob"}}}Justbe really really carefulhere not to trust the schema or data given to you by the clients. Basic web security 101 is that all data from the client is untrustworthy. This means that your schema is only as good as allowing the client to validate the data passed to it from the server ( a presumed trusted source) and validate the user input. You would always AGAIN validate the user input on the server side using a private schema that can not be modified by the clients.Without knowing the specifics of you application, that this is all probably over egging the problem it a bit. Typically configuration files are made to be forward compatible with firmware. So the definition of "name" is fixed by some agreed standard (without a digital schema) and both ends (server & clients) just use that standard. Later, if you need to, you would introduce a new "user_2020" and depreciate the old "name" entry.I'm not a fan of config versions numbers in the config file because that just creats a brick wall for old firmware versions to hit at some point.How out-of-sync do you expect the firmware versions between all devices to be anyway?I generally think it's better to give a lot of consideration to how 1) a conf file is structured and the minimum non changing variables that will at least ensure the device is always able to connect to the network and be upgraded and 2) how and when you are going to distribute firmware releases to the clients. Do this now rather then going to the effort to build in mechanisms to make it arbitrary.

author: krho
author_rank: Regular Contributor
timestamp: March 19, 2018, 08:07:24 am
content: If you need something that' easy to parse on MCU and desn't require larger amounts of memory you can look at CBOR. This is used more and more in embedded for multitude of things. For example the hardware 2FA USB keys use it.

author: Jeroen3
author_rank: Super Contributor
timestamp: March 19, 2018, 08:33:33 am
content: XML is widely supported, I'd use XML.Besides, XML has inline schema, which is what you want, right?Json is just a method to move data imho.Ini is a quick config next to an executable, not for anything fancy. It's awful.

author: css
author_rank: Newbie
timestamp: March 19, 2018, 09:24:58 am
content: Hi allThanks a lot for all your inputs.rhb, thanks the the suggestions on pain ascii. I would, if possible, like to avoid creating a custom format.ejeffrey, we currently use INI file format and are not happy with it - for the same reasons as you point out. I agree that a solution could be to implement a subset of eg. JSON. I would like if the server does not need to know the restrictions - as it will need to know this for every firmware version, which will likely be a pain to maintain. The restrictions could be added in the eg. JSON structure - but this would be a custom interpretation. Ideally, I would like the retrictions to be formulated in a standard way - such that it can be loaded into every existing program supporting this.Dubbie, thanks for the link to the parser. Will definately give this a try if we end up with JSON.sleemanj, thanks for the suggestion. Again, the weak point with JSON in regards to this specific application is the lack of a standard way to define restrictions to parameter values.David Chamberlain, thanks for the example. It is definately a possibility to add the schema in the top of the file. The "problem" is that 3rd party software will not be able to recognise this (i assume). In this case, it is really the client (the embeded device) which knows what the correct schema is. The server merely provides a interface to edit the config files and should ideally not know any detail of the content (thats why I need the config file to be completely self-explanatory). The "edited" config file will be validated by the embedded device before it is loaded.krho, Thanks for the suggestion on CBOR. We have looked into this. The format itself is interesting and seems suitable for embedded devices (which JSON, XML, YAML etc. are really not). However, to my knowledge, CBOR does not specify any "schema" mechanism.Jeroen3, Thanks for the suggestion. I was not aware that XML supported inline schema - I will definately look into this. Only problem with XLM is that it is not the simplets to parse on a small embedded device in ANSI C.Best regardsChrstian

author: Jeroen3
author_rank: Super Contributor
timestamp: March 19, 2018, 09:42:47 am
content: Quote from: css on March 19, 2018, 09:24:58 amXML is the most suitable what you want from it. But it might not fit your platform.There is libxml, but I think you'll be much more capable with XML on C++.How many settings do you need to store? Since xml has some overhead, does the device have external memory?I did once consider this myself and had anINI file ideawhere the line above the value contains description, limits and type.But since developing and maintaining IoT devices with one single developer is not quite doable, I had to drop this project. Customer would have to be able to edit this file, event without editor.Recently I developed a different project, with future compatibility (I hope) that just has an XML file describing the struct in the embedded device with name, offset, type, min, max and description, and a blob going back and forth.I did create a function in the embedded device that is capable of generating this XML from it's "parameter properties" data-set (big nested struct). Excluding the description, memory is limited.This can be imported in a configuration tool, and edited as a big table of parameter (100-ish).It's A LOT of work. The parameter code has the same LOC than the core functionality of the device. Ridiculous.

author: css
author_rank: Newbie
timestamp: March 19, 2018, 09:59:22 am
content: Hi Jeroen3, thanks for the reply.The device has some external memory which can be used to store the file - as such the file size it not really an issue.I share your concern that selecting XML will be quite a big task on the embedded side. JSON (or a similar binary alternative) is properly much simpler, with the downside that the "schema" / restrictions will have to be included in the file in a "non-standard" way.

author: hamster_nz
author_rank: Super Contributor
timestamp: March 19, 2018, 10:19:53 am
content: Anything that is powerful enough to be fully descriptive is going to be painful when it comes to trust and the update cycle.Why not keep you config file as simple as possible, and allow it to be validated against a schema you have trust in (e.g. because it is an immutable part of the firmware)Otherwise you leave a way to inject unexpected crap into your system.- Deeply nested structures to upset your parser- Really long data strings to blow out buffers- Strings with magic meta-characters in it (e.g. NULLs)- Internationalization problems when people put accented character in fields.- Range/type errors- Unexpected conflicts, where you assume that the schema will prevent incompatible options being selected- Screwing around data types- People providing 'alternative' schemas- Memory leaksIt should only take a short while to code reading in a config parser with a really simple syntax like:# commentoption[.suboption]*:valueAnd then supplying the schema as a separate resource that can be used to generate, then validate it before trusting it.If you need it to be a single human editable file, provide the metadata as passive comments that can be just ignored during the update.

author: css
author_rank: Newbie
timestamp: March 19, 2018, 10:36:06 am
content: Hi hamster_nz, thanks for your reply.I dont think trust is an issue in this setup. Let me try to explain in more detail.The embedded nodes carry the configuration. The nodes "knows" the restrictions of each configurable parameter. On boot, the embedded device loads the configuration file, and validates every entry against the restrictions stored in internal flash. As such, the validation of the configuration file does not rely on the schema.What I seek is a way toexportthe parametersandrestrictions - such that the configuration parameters and restrictions can be presented to a user. By including the restrictions in the configuration file, the user can make sure to not enter eg. invalid parameter values (if he does anyway, then the configuration is just rejected by the embedded device, which will use a default safe value).By including the restrictions (and possibly also descriptive strings), the tool used for changing the configuration (in this case a web interface) does not need to know anything about the configuration file - it can simply load the current values and, from the schema, present the user the valid selections for each parameter.I hope above is clear.

author: css
author_rank: Newbie
timestamp: March 19, 2018, 10:38:10 am
content: Found this schema description for JSON:http://json-schema.org/Seems to be a draft specification. Also, the it seems not possible to include the schema "inline" in the configuration file (I may need to accept that the schema is stored in a separate file)Using JSON + JSON Schema, this web framework seems very interesting and may be a solution to the "web" part of the challenge:https://github.com/mozilla-services/react-jsonschema-formWith interactive example:https://mozilla-services.github.io/react-jsonschema-form/

author: rhb
author_rank: Super Contributor
timestamp: March 19, 2018, 12:31:22 pm
content: Just to be sure you understand my point. Once you embrace something like JSON, the level of complexity goes up a lot. A major problem for the project I worked on was attempting to resolve was conflicting schema when moving data between different software packages. Should you *ever* need to make a major change to your schema you will be facing an epic headache. So you need to ask yourself, "Do I feel lucky?"Yes, a flat file database such as I suggested is not as "elegant". But unless you have a very complex configuration problem, it's much easier to support. At the configuration file level, you're requiring the support staff to read and understand JSON syntax if there s a problem with the web based UI output. Remember KISS.

author: technix
author_rank: Super Contributor
timestamp: March 19, 2018, 01:36:09 pm
content: Throwing in my random 2 cents: SQLite, if you like a binary format. JSON is still my best option since it is a lightweight text based representation, and it can be handled easily, however adding a schema can require some creativity. XML does have schema support, but the parser is very heavyweight. SQLite is kind of the middle ground here, with a moderately heavyweight parser. Also being a binary format it might work better on embedded systems, as instead of processing the file as a data stream and store the parsed data elsewhere, the SQLite database file can be mapped directly into RAM (for example, using QSPI) and libsqlite can be instructed to operate directly on the memory mapped database file without copying data first.