creating a CAN open dictionary - Page 1
Forum section: electronics / general

author: Simon
author_rank: Unknown
timestamp: January 22, 2025, 05:06:38 pm
content: I'm trying to set up a CAN open dictionary.There is more than one way of doing any one thing so I am at several cross roads.CAN open objects can be any type. The ones I have dealt with so far have ranged from 8 bit signed to 32 bit unsigned, although no value ever seems to exceed 2 billion (2^31). Should I simply use signed 32 bit variables for everything to simplify the management or is it best to use the actual type.The other aspect that I am unsure about is do I manage each index as an array of its sub indexes or just treat everything the same, effectively ignoring the 16 bit index and 8 bit sub index and simply refer to any object by the combined 24 bit "address".Naturally I cannot create 2^24 variables, even if I were on a computer I would need over 8GB of memory for a full 128 device system, OK this is an extreme example but it shows that some other means of identifying each object is necessary even on a computer and I am working on a micro controller at the moment but this will port across to a computer as well.So if I create each variable as an int32 I could simply create an array for each index that contains all of its sub index values. I then would not need to store information about each sub index. A this point I can create an array of pointers which list the address's of the indices.I would have the further problem of dealing with multiple nodes which I expect is just another level to the array of values. Different devices have different index values for their specific functions but they would have in common indices that relate to the communication setup for example so I expect that I end up with several arrays each covering different ranges of indices.

author: Nominal Animal
author_rank: Super Contributor
timestamp: January 23, 2025, 12:22:49 am
content: I'd say you're thinking about it on a too abstract level. Consider the following questions:How many entries will your object dictionary have? The maximum is 65536. Most profiles have maybe a dozen to a few dozen.How many entries are basic/scalar/non-indexable, and how many are composite (arrays, records, strings)? Only the composite ones can be 8-bit indexed.Are the entries known at compile time, or do you need to create and destroy entries at run time?The more general you want your approach to be, the more severe compromises and allowances your code has to make.The number of entries in a dictionary indexed by a 16-bit (or 24-bit, if you happened to have a dictionary with only record objects) unsigned integer defines whether you should use a simple unsorted array (small number, say a dozen or two entries) or a sorted array (binary search yielding the entry in N+1 simple steps when the number of entries is 2N) or a hash table. If the number of entries is known at build time, you can use a perfect or near-perfect hash function, by brute force searching among configurable hash functions and hash table sizes.In any case, let's assume the index search provides you with aslot, describing the entry found. What should that slot contain? (We assume the index search already verified the slot corresponds to the searched-for index.) The Wikipedia articledescribes the six properties defining each, but those are conceptual; instead consider what you actually need in practice. (I myself write small experimental programs, usually in Linux, to test. You might wish to write a program implementing the desired CANopen device or functionality, simply to find out. I've found that this actuallysavesoverall development time, because of much better understanding of the problem domain at the practical implementation level.)The answer I would implement, depends exactly on the set of basic/scalar and composite entry types you have. I would be tempted to use an array each for each basic type, strings, and arrays (of each basic type), so that the entry identifies the type and the index in the value array. (Thus, stuff like 8-bit temperature might be atobjects_8bit, which you'd probably #define asOBJECT_TEMPERATUREfor use in the firmware code outside the object dictionary.) For immutable strings, the value array entries contain a pointer and the length; for mutable strings, a pointer (to RAM), current length, and maximum length; for arrays, a pointer, current length, and maximum length. Composite or structure types are the most complicated to support, so I would carefully investigate those before considering any implementation. I suspect I would use an array for all composite/structure types, each element having a pointer to the data, and a fixed-size type array (say 128 bits, with 4 bits per member type, for a maximum of 32 members per structure).Looking back, IsuspectI would use an array of sorted 32-bit elements for the object lookup. The high 16 bits of the entry would be the object index, and the low 16 bits would identify the array and the index in the array. The size of this array is determined by the number of objects (not counting sub-indexes), at most 65536 entries (262,144 bytes). A binary search will find the index or determine nonexistence in 1+log2Nsteps, i.e. at most 17 steps. It's just comparisons, additions, bit shifts, and (conditional) jumps, so this should be sufficiently efficient, not requiring an even faster hash table approach (which would be typically a single step only, but that containing more machine operations).How to divide the 16 bits into index and type depends on the number of objects of each type. Because there are at most 2¹⁶ objects, thiswillsuffice; the worst case is each type in an arbitrary consecutive range of indexes, determined at build time.Let me know if you need some example code to illustrate the above. Again, personally, I would simply write some test code to run on a fully-hosted system (I use Linux, but anything will work) to determine the properties of each approach. (Some I have already done; for example the sorted array with unique ID in the most significant bits. You want to use most significant and not least significant bits, because since the ID is unique, the values can be compared without masking when most significant bits are used. If you used least significant bits, masking (AND) would be required.)If you don't want my walls of text to your questions, just drop me a PM, and I won't bother you with mine any further.

author: Simon
author_rank: Unknown
timestamp: January 24, 2025, 09:24:12 am
content: Well I welcome any view points. My exposure to CAN Open is limited and what I am trying to avoid is working something out only to discover later a new aspect I need to account for that has me rewriting everything.I suppose it is worth looking at this from the separate perspectives of a master and a slave device and possibly constructing the dictionary differently for each. My initial use is a 48MHz M0+ micro controller so I have a reasonable amount of grunt but nothing like a SBC.Dealing with the library will be at it's toughest when doing SDO communication and I have only had to write this from the masters side so far. Initially I wouldsequentially search each index and sub index until I found a match, took a while on 50 odd items, I did put the ones I know I used a lot at the start of the list.The master controls the SDO communication, the master sends a message with an index and a sub index and should not send another until there is a response from the slave or it times out. This means that an SDO message from the slave should be for the same index and sub index on request only, so there is no longer a need to search for the related object. So I make a note of which one I send a request for and check that the response is for the same one or it's an error, much, much faster. If I don't need to do a search then I can implement the library in a way that is easier to write code with and works well with having to handle multiple nodes of the same type that will have identical copies of dictionary.From the slave point of view I have no idea what I am likely to design. How many objects I will actually need and maybe a not so idealized solution is best to just accommodate the few I would use. I would preferably use PDO communication and providing my objects do not exceed 32 bytes I can send them all as PDO messages. This is what I am currently doing. If I confine index and sub index searching to the slave side only I can set up a slave dictionary that is easier to search and perhaps slightly more convoluted in design but that works efficiently for searches.What I will end up with is systems that are a combination of my master which currently runs on 48MHz of M0+ with 16kB of RAM (could go to 32kB) controlling up to 2 off the shelf motor drivers and other items of my own design that I am making at the very least CAN Open compatible in terms of being able to operate on the same bus without issues.

author: Nominal Animal
author_rank: Super Contributor
timestamp: January 26, 2025, 08:18:57 pm
content: Quote from: Simon on January 24, 2025, 09:24:12 amAgreed. I guess my most similar device is Kinetis MKL26Z64 (Teensy LC, Cortex-M0+, 48MHz, 8k RAM, 62k Flash), which has a single-cycle 32×32=32-bit truncating multiplication (low 32 bits of result only). I assume you'll avoid floating-point types for now?For anyone else interested:The ARMv6-M architecture does not support unaligned 16-bit (loading from odd addresses) or unaligned 32-bit addresses (loading from addresses not a multiple of four), which means memory accesses and buffer alignment (like for the SDO functions) has to be carefully considered. 8-bit signed and unsigned values are trivially extended to 16 and 32 bits (single instruction), and 16-bit to 32 bit (ditto). When loading from RAM, 8-bit and 16-bit values are zero- or sign-extended to 32 bits. (These are implemented in C with simple casts to/fromuintN_t/intN_ttypes as needed, the compilers will optimize these.) The architecture reference manual ishere.Quote from: Simon on January 24, 2025, 09:24:12 amJust for experimentation, I thought about how to efficiently pack and unpack the CANopen composite types to/from the corresponding C structures. Here's what I came up with:Code:// SPDX-License-Identifier: CC0-1.0// Original author: Nominal Animal, 2025#include <stddef.h>#include <stdint.h>// Supported mapping types:#define BOOLEAN_IN_8 MAPPING_BOOL_8#define BOOLEAN_IN_16 MAPPING_BOOL_16#define BOOLEAN_IN_32 MAPPING_BOOL_32#define UNSIGNED8_IN_8 MAPPING_8_8#define INTEGER8_IN_8 MAPPING_8_8#define UNSIGNED8_IN_16 MAPPING_U8_16#define INTEGER8_IN_16 MAPPING_I8_16#define UNSIGNED8_IN_32 MAPPING_U8_32#define INTEGER8_IN_32 MAPPING_I8_32#define UNSIGNED16_IN_16 MAPPING_16_16#define INTEGER16_IN_16 MAPPING_16_16#define UNSIGNED16_IN_32 MAPPING_U16_32#define INTEGER16_IN_32 MAPPING_I16_32#define UNSIGNED24_IN_32 MAPPING_U24_32#define INTEGER24_IN_32 MAPPING_I24_32#define UNSIGNED32_IN_32 MAPPING_32_32#define INTEGER32_IN_32 MAPPING_32_32#define REAL32_IN_32 MAPPING_32_32// Note: In the object, the least significant bit of the byte defines the state.// In the variables, standard C logic (0=false, nonzero=True) is used.// 4-bit encoding for the mapping types:#define MAPPING_NONE 0 // Object Variable#define MAPPING_BOOL_8 1 // BOOLEAN U8/I8#define MAPPING_BOOL_16 2 // BOOLEAN U16/I16#define MAPPING_BOOL_32 3 // BOOLEAN U32/I32#define MAPPING_8_8 4 // U8/I8 U8/I8#define MAPPING_16_16 5 // U16/I16 U16/I16#define MAPPING_24_24 6 // U24/I24 U24/I24#define MAPPING_32_32 7 // U32/I32/R32 U32/I32/R32#define MAPPING_U8_16 8 // U8 U16/I16#define MAPPING_U8_32 9 // U8 U32/I32#define MAPPING_U16_32 10 // U16 U32/I32#define MAPPING_U24_32 11 // U24 U32/I32#define MAPPING_I8_16 12 // I8 I16#define MAPPING_I8_32 13 // I8 I32#define MAPPING_I16_32 14 // I16 I32#define MAPPING_I24_32 15 // I24 I32// Union type for defining mappings; up to 8 members per mappingtypedef union {uint32_t all;struct {uint32_t member1:4;uint32_t member2:4;uint32_t member3:4;uint32_t member4:4;uint32_t member5:4;uint32_t member6:4;uint32_t member7:4;uint32_t member8:4;} members;} mapping;#define DEFINE_MAPPING(Name, ...) \const mapping Name = { .members = { __VA_ARGS__ } }#define DEFINE_LOCAL_MAPPING(Name, ...) \static const mapping Name = { .members = { __VA_ARGS__ } }// Object and variable size lookup array.// Note that the order of the elements does not necessarily match the list.// High nibble is the object size, low nibble is the variable size, for each type.const uint8_t mapping_lookup = {[ MAPPING_NONE ] = 0x00,[ MAPPING_BOOL_8 ] = 0x11,[ MAPPING_BOOL_16 ] = 0x12,[ MAPPING_BOOL_32 ] = 0x14,[ MAPPING_8_8 ] = 0x11,[ MAPPING_16_16 ] = 0x22,[ MAPPING_24_24 ] = 0x33,[ MAPPING_32_32 ] = 0x44,[ MAPPING_U8_16 ] = 0x12,[ MAPPING_U8_32 ] = 0x14,[ MAPPING_U16_32 ] = 0x24,[ MAPPING_U24_32 ] = 0x34,[ MAPPING_I8_16 ] = 0x12,[ MAPPING_I8_32 ] = 0x14,[ MAPPING_I16_32 ] = 0x24,[ MAPPING_I24_32 ] = 0x34,};// Return the number of bytes in the packed object.uint32_t mapping_objsize(mapping m) {uint32_t size = 0;while (m.all) {size += (uint32_t)(mapping_lookup[ (m.all << 28) >> 28 ]) >> 4;m.all >>= 4;}return size;}// Return the number of bytes in the unpacked C structure.uint32_t mapping_varsize(mapping m) {uint32_t size = 0;while (m.all) {size += ((uint32_t)(mapping_lookup[ (m.all << 28) >> 28 ]) << 28) >> 28;m.all >>= 4;}return size;}// Pack variables from 'src_var' to object 'dst_obj' using mapping 'm'.void pack(void *dst_obj, const void *src_var, mapping m) {unsigned char *dst = dst_obj;const unsigned char *src = src_var;while (m.all) {switch ((m.all << 28) >> 28) {case MAPPING_BOOL_32: *(dst++) = !!(src | src | src | src);src += 4;break;case MAPPING_BOOL_16: *(dst++) = !!(src | src);src += 2;break;case MAPPING_BOOL_8: *(dst++) = !!(*src++);break;case MAPPING_32_32: *(dst++) = *(src++);case MAPPING_24_24: *(dst++) = *(src++);case MAPPING_16_16: *(dst++) = *(src++);case MAPPING_8_8: *(dst++) = *(src++);break;case MAPPING_U8_16:case MAPPING_I8_16: *(dst++) = *src;src += 2;break;case MAPPING_U8_32:case MAPPING_I8_32: *(dst++) = *src;src += 4;break;case MAPPING_U16_32:case MAPPING_I16_32: *(dst++) = src;*(dst++) = src;src += 4;break;case MAPPING_U24_32:case MAPPING_I24_32: *(dst++) = src;*(dst++) = src;*(dst++) = src;src += 4;break;case MAPPING_NONE: break;}m.all >>= 4;}}// Unpack object 'dst_obj' into variables at 'dst_var' using mapping 'm'.void unpack(void *dst_var, const void *src_obj, mapping m) {unsigned char *dst = dst_var;const unsigned char *src = src_obj;while (m.all) {switch ((m.all << 28) >> 28) {case MAPPING_BOOL_32: *(dst++) = *(src++) & 1;*(dst++) = 0;*(dst++) = 0;*(dst++) = 0;break;case MAPPING_BOOL_16: *(dst++) = *(src++) & 1;*(dst++) = 0;break;case MAPPING_BOOL_8: *(dst++) = *(src++) & 1;break;case MAPPING_32_32: *(dst++) = *(src++);case MAPPING_24_24: *(dst++) = *(src++);case MAPPING_16_16: *(dst++) = *(src++);case MAPPING_8_8: *(dst++) = *(src++);break;case MAPPING_U8_16: *(dst++) = *(src++);*(dst++) = 0;break;case MAPPING_I8_16: dst = src;dst = ((uint32_t)(int32_t)(int8_t)(src)) >> 7;src += 1;dst += 2;break;case MAPPING_I16_32: dst = src;dst = src;dst = dst = ((uint32_t)(int32_t)(int8_t)(src)) >> 7;src += 2;dst += 4;break;case MAPPING_I24_32: dst = src;dst = src;dst = src;dst = ((uint32_t)(int32_t)(int8_t)(src)) >> 7;src += 3;dst += 4;break;case MAPPING_U8_32: *(dst++) = *(src++);*(dst++) = 0;*(dst++) = 0;*(dst++) = 0;break;case MAPPING_U16_32: *(dst++) = *(src++);*(dst++) = *(src++);*(dst++) = 0;*(dst++) = 0;break;case MAPPING_U24_32: *(dst++) = *(src++);*(dst++) = *(src++);*(dst++) = *(src++);*(dst++) = 0;break;case MAPPING_NONE: return;}m.all >>= 4;}}Note the license: you can freely use this code or modify it however you want, even in proprietary codebases. No need to credit me either; I consider this to be in Public Domain (per Creative Commons Zero 1.0).The idea is that we use a 8-element 4-bit vector packed in a 32-bit unsigned integer (mappingtype) to specify the mapping between the CANopen objects and C variables/structures. It is limited to BOOLEAN, INTEGER8, INTEGER16, INTEGER24, INTEGER32, UNSIGNED8, UNSIGNED16, UNSIGNED24, UNSIGNED32, and optionally REAL32 CANopen basic types. See the aboveCANopenBasicType_IN_SizeInBitsmacros.Let's say you have aSTRUCT OF UNSIGNED8a, INTEGER32b, UNSIGNED16c, and INTEGER8ddefined asname. One corresponding properly aligned C structure would bestruct {uint32_ta;int32_tb;uint16_tc;int16_td;}name_var;and the mapping between these two would beconst mappingname_map = { .members = {UNSIGNED8_IN_32,INTEGER32_IN_32,UNSIGNED16_IN_16,INTEGER8_IN_16,};To pack the values inname_vartounsigned char buffer[], you dopack(buffer, &name_var,name_map);To unpack the values toname_varfromconst unsigned char buffer[], you dounpack(&name_var, buffer,name_map);The helper functionmapping_varsize(name_map)returns the size of the variable (C structure) defined byname_map, in bytes, andmapping_objsize(name_map)the size of the packed object in the buffer, in bytes.This allows two completely different implementation approaches for the CANopen object dictionary.One is where the dictionary stores the values (in 32-bit or 64-bit units in RAM) in the packed transmission format, and code callspack()to update the values.The other is where the dictionary only knows about the C structures describing that data, and the mapping of course. To send an object, it callspack()to obtain the packed object from the C structures; and when it receives an object, it callsunpack()to update the C structures. (Each object then has a 32-bit fixed mapping, and a pointer to the beginning of the related C structure.)The code includes some idioms that help GCC 9.2.1 generate better code for Cortex-M0+ (ARMv6-m).In particular, to pick thenleast significant bits ofv,((uint32_t)(v) << (32 -n)) >> (32 -n)tends to generate better code (just two shifts) than masking.

author: Simon
author_rank: Unknown
timestamp: January 28, 2025, 10:26:02 am
content: Well if unaligned access to variables is not possible on the architecture I assume that something stored in a structure out of alignment will either get aligned by the compiler padding out the structure or take multiple operations to access the two word aligned locations, dissect them and put the relevant parts back together at a great cost to speed. If that is the case I may as well use int32 only as I currently do but get cleverer at how they are organise so that sub indices can be located without searching.If all variables are actually int32 then it does simplify things.An array of sub index values is easily created, An array of pointers to the sub index arrays represents the object indices. A parallel array of index numbers would allow the program to search for the array index that corresponds to an object index number.That all works fine for the slave that has to go looking for the variable referenced by the index and sub index number in a message. The only actual search required is for the index. Of course this does not take into account the actual size of an object or any other properties that I suppose a slave device should check against before just putting the data into the variable on a write request.so if a structure was created that defined all of the properties of a sub index object this could contain the int32 variable itself. As everything is stored as int32 only one such structure type is required and an array of these can form the sub index array. Everything is declared in one place.This still means that two arrays must be set up, one for the pointers to the arrays of sub indices and another to list the index number in each array location. These two could be combined into a single structure that would have the pointer and the index number and array of this type made.This seems fairly straight forward on a slave in terms of finding data by index and sub index. The main program can refer directly to the value in the sub index structure by structure name.Ah but this all means that the whole structure must sit in RAM, to try and have the non changing data kept in the program flash the object value would have to be replaced with a pointer. I suppose that further simplifies normal use of the variable as it is just a variable and all of the pointery malarkey only happens in predefined functions that send and receive the object over the CAN bus. Also the variable can be of the correct type which might be helpful for PDO communication which is the bit that should be the most efficient.For a master however there is a slight complication in that several nodes will have the same object type, I suspect that this is best resolved with an array of the same objects although this means that I need to map each node to an index to avoid redundant slots. For each node type a different setup will be required with a note of how many devices of this type there are. It would of course be silly to duplicate the common stuff like device type and comms setup so I would probably have to have a common dictonary for the comms data for all devices and then a device specific dictonary with pointer mapping in some arrays to direct data for each node ID.Does that all sound nuts?

author: Nominal Animal
author_rank: Super Contributor
timestamp: January 28, 2025, 05:03:49 pm
content: Quote from: Simon on January 28, 2025, 10:26:02 amDo you need to be able to add and delete objects in the object dictionary, only modifying theirvalues; or do you need to construct the object dictionary at run time?One approach to solve many of the problems you outlined would be to use overlappabledictionary fragments, and construct each dictionary from one or more fragments in the search order, perhaps as an array. Basically, each fragment would be described by the lowest index in the fragment, the highest index in the fragment, the type of the fragment (more on this), a pointer to its index array, a pointer to its definition (object types, access modes, et cetera), and a pointer to its data. You could have different fragment types for VAR indices (those with the value stored in sub-index 0) - or even eight different types: one each for strings, 8-bit, 16-bit, and 32-bit VAR objects in RAM; and the same for those in Flash (that would automatically haveconstaccess mode). Then, the data would be as compact as possible. Some fragments could be in Flash, and some in RAM. Fragments with identical structure and type but different values could refer to same index arrays and definitions, but different data.To look up an object or sub-index in an object dictionary, one would check each fragment in it. Only when the index is within the index range, would one do a binary search within the actual index dictionary. If not found, then the other fragments would be checked. This way, object dictionaries containing the same entries could be stored in a single shared fragment. (If objects can be created and deleted at run time, it can get hairy, though.)Arrays and records only differ in that for arrays, all elements have the same data type, whereas in records each element has their own data type. Both use sub-index 0 for the number of elements. In both cases, each element has their own access mode (2 bits). The maximum number of elements is actually 254, not 255 as I wrote earlier, because sub-index 255 is also special, but not required to be used. It is really only records that need to use fixed-allocated-size elements (say, 32 bits), and that only so that it is fast to look up the value of a specific sub-index. If the size were to vary, you'd need to iterate through the types from the first onwards, to determine the offset to the target sub-index.If you combine arrays and records, instead describing them all as record8/record16/record32/record64, depending the size of each element, and additionally use a byte per element to record the actual data type and access mode, memory use would be minimal, and sub-index lookup constant-time for both. You could save 6 bits per element if you implement array access separately, but I doubt it is worth the savings.

author: Simon
author_rank: Unknown
timestamp: January 29, 2025, 09:29:13 am
content: My plan is to set everything up in the program at the time of writing, no run time edits. I am afraid that you have rather lost me.What I was thinking was that in order to find an object at sub index level I have to first locate it by index, then I can go by sub index. Sub indices seems to be consecutive, if I am using one of the sub index objects I may as well set up the entire index as I am likely using more than just the one. This makes it possible to pin point a sub index object within the index if I were to have an array of sub index descriptions for each index, this allows me to vary the array length for each index and not waste space.So if I can just go straight to a sub index of any index once located the next problem is to find the index, these I can't just put in an array starting at the lowest index used up to the highest. The best I can do there is to have an array that I can search through easily using the array index.From the slaves point of view I would have A structure type that describes each index, it's index number and a pointer to sub index definitions if there are more than one or simply to the value if there is only one. This has to be searched until the correct index number is found.If the index description points to an array of sub indices this array would be an array of structure types that describes the sub index with a pointer to the variable in memory.For slaves this is all fine each is unlikely to change.I'm wondering if what you are talking about is specific structures of various arrays that describe some range of indices and their sub indices that cover specific ranges, perhaps with each pointing to the next mini dictionary.One thing I have struggled on is how to initialize structures when I am setting them up. At the moment I create the structured type even where this will only have one instance and then create a function that fills all of the structure out at the start of the program. I am sure that this is optimized out but it kind of feels clunky.From a masters point of view the 3 main sections of object dictionary could be divided up. The coms section (0x1000 - 0x1FFF) will be the same across all nodes, so a single array of this range works. Then you have the profile specific section (0x6000 - 0x9FFF) that should be separate as this is quite reusable as well. Then there is the (0x2000 - 0x5FFF) range that is specific to the manufacturer.Each section can be kept as separate sub dictionaries. On message reception the index number range would be identified so that the correct mini dictionary can be interacted with, this way each part of the dictionary is created just as big as necessary. Given how manufacturers often replace much of the profile specific section with their own manufacturer specific section it may be just as well to have a dictionary per device model number.So the master would be able to use the same structure as the slaves but the actual variables in ram would be arrays, A small array can map each node ID to the correct array index in each of the dictionary arrays.

author: DiTBho
author_rank: Super Contributor
timestamp: January 29, 2025, 11:24:38 am
content: Doesn't Bosh or Magneti Marelli provide any guidance line?

author: Simon
author_rank: Unknown
timestamp: January 29, 2025, 12:18:36 pm
content: Why would they? Bosch designed CAN bus which is already at a higher level of standardization than your typical electrical spec only bus such as RS485 but they did stop at here is a bus system that will transmit up to 64 bits per message, do with it what you will.Both may well be members of the CAN Open consortium, but any implementation they have will be their own internal IP for their own CAN open products.The CAN bus controller on my micro controller is designed by Bosch, it is the only controller I have ever used so I can't compare but it looks like it was designed with CAN Open in mind.

author: Nominal Animal
author_rank: Super Contributor
timestamp: January 30, 2025, 03:46:52 am
content: Quote from: Simon on January 29, 2025, 09:29:13 amI explained it poorly, because I was still thinking it up at the time.Quote from: Simon on January 29, 2025, 09:29:13 amYes, except that the structures are in RAM and form an array, not a linked list.Instead of a single dictionary, you can use a sequence of dictionaries, looking up the index in each, one at a time, until found. (This kind of construction is often calledoverlay, but I chose to call itfragment, because of the inherent structure of the indexes, as I describe below.)Eachfragmentcan be used in more than one dictionary, and the change of value in one will be reflected in all dictionaries using the same fragment.At the face of it, this sounds like a lot of wasted computation, but it is not: it turns out that for typical implementations, each fragment will basically define a range of indexes, that rarely overlaps with other fragments' indexes.If one uses a different dictionary each for DEFTYPE, DEFSTRUCT, VAR, ARRAY, and RECORD objects, they will overlap, but implementation becomes much more efficient. In fact, it turns out that it makes sense to implement ARRAY objects as RECORD objects that just all happen to have all fields the same type. In practice, this means that at to look up any index, one will check at most two dictionary fragments. With binary search, it should not be too costly at all.Quote from: Simon on January 29, 2025, 09:29:13 amYes. Similarly, indexes are rather tightly clustered and not at all random.For example, all indexes below 0x1000 are definition objects (and there are none in the range 0x0260..0x0FFF. 0x0060..0x025F and 0x6000..0x9FFF are used by the eight logical device profiles. As a "manufacturer", you'll put your profile stuff in 0x2000..0x5FFF. Network variables are in 0xA000..0xAFFF, and system variables in 0xB000..0xBFFF. Mostly, the accesses will be to communication profile area, in 0x1000..0x1FFF. These are not randomly filled, either, and tend to be densely filled in the lower indexes, with an occasional hole here and there.DEFTYPE objects, when read, simply provide the length of the type defined, in bits, as an UNSIGNED32, or 0 if variable. These are all also read-only or const, so all you need is 8 bits per index. DEFSTRUCT objects are slightly more complicated, as their sub-index 0 indicates the number of fields in the struct as an UNSIGNED8, and the following sub-indexes the UNSIGNED16 field type. In practice, each index can just refer to an array of UNSIGNED16 values. It turns out that these can easily be packed into a single array of UNSIGNED16, and each object index just refer to the offset in this array. Thus, these are rather easy.VAR objects are those that have the data field in sub-index 0, and use no other sub-indexes. RECORD objects have up to 254 fields of varying types, and ARRAY objects up to 254 fields of the same type; both having the length as UNSIGNED8 in sub-index 0. These are the ones you'll most often be accessing.When looking up a field via index:sub-index, there are several pieces of information you end up needing:The address to the field itselfNumber of bits/bytes used by the valueType of the value (BOOLEAN, UNSIGNED, INTEGER, REAL, VISIBLE_STRING, OCTET_STRING, UNICODE_STRING)Access mode (const, read-only, write-only, read-write)Annoyingly enough, ARMv6-m can return a 64-bit integer in registers R0 and R1, but will not do so for 64-bit unions or structures. So, one ends up having to do something likereturn ((fieldspec){ .ref =addr, .size =size, .datatype =type, .access =access}).packed;const fieldspec field = { .packed =find_field_using_index_and_subindex() };withCode:typedef union {uint64_t packed;struct {union {void *ref;uint32_t *u32;uint16_t *u16;uint8_t *u8;int32_t *i32;int16_t *i16;int8_t *i8;uint32_t *boolean;char *visible_string;unsigned char *octet_string;uint16_t *unicode_string;};uint16_t size;uint8_t datatype:6;uint8_t access:2;};} fieldspec;so that you can use a single function to find the field, whether you are getting or setting the value, instead of having to copy-paste much of the same code. Such a function can easily support all different dictionary fragment types, too, not only doing the binary search within each fragment that contains indexes in the target range, but also scanning through all fragments belonging to a "full dictionary". (The access mode is rather important, if you do not want a CANopen SDO message trying to modify a data field in Flash to crash your MCU.)The downside is that their initialization in C isannoying, because the description of a single object is split across multiple arrays. Generating the C code for the structures from a definition, however, is simple. I don't likecodegeneration at all, but fortunately these are just structures and arrays in Flash and RAM, plus macros to define the names of entries in RAM so you can access the value directly also (if you want).For example, the VAR dictionary consists of three arrays (that would be members of a structure, here shown separately):const uint16_t var_index;const acctype var_acctype;field32 var_value;withCode:typedef union {uint8_t acctype;struct {uint8_t datatype:6;uint8_t access:2;};} acctype;typedef union {uint32_t u32;uint16_t u16;uint8_t u8;int32_t i32;int16_t i16;int8_t i8;uint32_t boolean;char *visible_string;unsigned char *octet_string;uint16_t *unicode_string;} field32;So that to initialize say index 0x1007, as the third element in the arrays, you end up having = 0x1007, //in thevar_indexarray initializer = { .datatype = DATATYPE_U32, .access = ACCESS_RO }, //in thevar_acctypearray initializer = { .u32 = 0 }, //in thevar_valuearray initializerplus a macro definition similar toSTATUS_REGISTER(dict) ((dict).var_value.u32)The ARRAY and RECORD dictionaries are even more annoying, because you have a variable number of fields per index. No problem at all if you generate the structure initializers, but damned annoying and error-prone if you maintain the initializers by hand. The implementation I'd look at combines these, using a similar arrays as for VARs, but with an additional offset into the acctype and value arrays per index, indicating where these start:const uint16_t rec_index;const uintN_t rec_start;const acctype rec_acctype[INDEXES+FIELDS];field32 rec_value[INDEXES+FIELDS];whereNis 16 if the number of indexes and fields (including sub-index 0) fits in 16 bits, 32 otherwise.To find a specific index:subindex, one does a binary search in therec_indexarray. If found ask'th element in that array, thenstart=rec_starttells where inrec_acctypeandrec_valuearrays the object starts at. The number of sub-indexes is always described by sub-index 0, i.e.rec_value.u8, so that will tell if the sub-index is within the object. If it is, thenrec_acctype[start+subindex]describes the data type and access mode, andrec_value[start+subindex]contains the field value itself. To define a preprocessor macro to a specific index:subindex, you use#defineMACRONAME(dict) ((dict).rec_value.datatype)allowing direct access to the field without any lookups.Because you need the two access mode bits per field, I see no sense in treating arrays and records differently; it only saves 6 bits per field, but duplicates a lot of common things. Instead, I think it makes sense to treat arrays internally as records that just happen to have all fields the same type (except sub-index 0 as UNSIGNED8).In all cases, instead of actual arrays, the dictionary fragment structures could use pointers, thus allowing the same fragment structure to be used with different values. Obviously, that makes hand-maintenance a nightmare, although with proper naming, generating the arrays and letting the user to construct the fragments as needed (via say helper macros), maintenance and use for us humans could be made quite nice and easy. After the underlying logic is explained, of course; without that, this definitely looks like black magic at the first glance.Quote from: Simon on January 29, 2025, 09:29:13 amExactly! While the C99 and later array initializers allow you to define array entries in a random order -- for example,Code:const unsigned char A = { 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 };const unsigned char B = {  = 0,  = 1,  = 2,  = 3,  = 4,  = 5,  = 6,  = 7,  = 8,  = 9 };defining the exact same array values –– each object (index definition) has to be spread into several different arrays; and modifying the length of one array or record will move the datatype, access, and value indexes for many other objects in the same fragments, making hand-maintenance of the initializers in C basically a fool's errand.If you use Awk (from a tabulated definition) or Python (from a tabulated, CSV, or structured/XML definition) to generate the initializers, then there is no problem. One can even add enough comments to make human verification/reading of the structures easy. With a Makefile dependency on the source definition on the generated files, it'd even build easily.My offer of example code still stands, but be fully aware of the abovementioned limitations/properties of my approach. I do claim the binary search to be extremely effective here, although I haven't microbenchmarked it on any real-world CANopen dictionaries (simply because I don't have any real-world profiles). Cache locality is far from perfect, but I suspect that on Cortex-M0/M0+ it won't matter that much.The way I implement binary search in this case, assuming indexes sorted in ascending/increasing order, in pseudocode isIf the sought-for index is less than the first index in this fragment, it is not within this fragment.If the sought-for index is the first index in this fragment, we found it at offset 0.If there are less than 1 index in this fragment, it is not within this fragment.Otherwise, setlast= one less than the number of indexes in this fragment.If the sought-for index is larger than the last index in this fragment, it is not within this fragment.If the sought-for index is the last index in this fragment, we found it at offsetlast.Do a binary search between (and excluding) offsets 0 andlast:Setoffset_min= 0 andoffset_max=last, then loop:Setoffset= (offset_min+offset_max) / 2, using 32-bit unsigned integer arithmetic to avoid overflow.Ifoffset==offset_min, it is not within this fragment.If the sought-for index is less than index atoffset, setoffset_max=offset.Otherwise, if the sought-for index is greater than index atoffset, setoffset_min=offset.Otherwise, the sought-for index is at offsetoffset.This is basically bulletproof. It does mean that the first and last entries in the index array are examined first (which is horrible for cache locality, but on simple architectures like ARMv6-m shouldn't make any measurable difference), but no offset is examined twice, nor are there any excess conditional checks that can be avoided. For example, the single-object check ensures thatlastwill never be negative, which could cause an access out of array bounds. If it is possible to have empty dictionary fragments, that does need an additional check up front. Exactly 1+⌈log₂N⌉ accesses are done for an array ofNelements; the exact number of comparisons depends on the data and/or the hardware architecture. On ARMv6-m, for a full < = > comparison, the data is loaded to a register, then the two registers are compared, followed by two conditional branch instructions, followed by the third case. Binary search is compact and simple, but does do rather many conditional jumps (which I believe are "cheap" on ARMv6-m, compared to say x86-64, where they are quite "expensive".)

author: Nominal Animal
author_rank: Super Contributor
timestamp: February 01, 2025, 07:27:52 am
content: Even though you didn't ask, and likely are not that interested in it, here is my basically complete implementation in less than two hundred lines of C:Code:// SPDX-License-Identifier: CC0-1.0#include <stddef.h>#include <stdint.h>typedef union {uint32_t u32;int32_t i32;uint16_t u16;int16_t i16;uint8_t u8;int8_t i8;uint32_t boolean;char *visible_string;unsigned char *octet_string;uint16_t *unicode_string;// float real32;const char *const_visible_string;const unsigned char *const_octet_string;const uint16_t *const_unicode_string;} value32;#define ACCESS_CONST 0#define ACCESS_RO 1#define ACCESS_WO 2#define ACCESS_RW 3#define DATATYPE_VOID 0#define DATATYPE_BOOLEAN 1#define DATATYPE_U8 2#define DATATYPE_U16 3#define DATATYPE_U32 4#define DATATYPE_I8 5#define DATATYPE_I16 6#define DATATYPE_I32 7#define DATATYPE_VISIBLE_STRING 8#define DATATYPE_OCTET_STRING 9#define DATATYPE_UNICODE_STRING 10#define DATATYPE_REAL32 11#define DATATYPE_CONST_VISIBLE_STRING 12#define DATATYPE_CONST_OCTET_STRING 13#define DATATYPE_CONST_UNICODE_STRING 14// 15..63 unusedtypedef union {uint8_t value;struct {uint8_t datatype:6;uint8_t access:2;};} acctype;typedef struct {uint8_t length:8;uint32_t offset:24;} offlen;#define FRAGTYPE_VAR 1#define FRAGTYPE_REC 2typedef struct {const uint16_t *index; // Sorted in ascending orderconst uint16_t last; // One less than the number of objects in this fragmentconst uint16_t fragtype:2; // Type of this fragmentconst uint16_t padding:14; // Any use for these?union {struct { // .fragtype == FRAGTYPE_VARconst acctype *var_acctypes; // Access mode and datatype for each objectvalue32 *var_values; // Value of the data in each object (sub-index 0)};struct { // .fragtype == FRAGTYPE_RECconst offlen *rec_offlens; // Offset and length of each recordconst acctype *rec_acctypes; // Record field datatypes and access modesvalue32 *rec_values; // Record field data values};};} fragment;typedef union {uint64_t value; // uint64_t is returned in registers even on ARMv6-m.struct {value32 *data; // Pointer to the data in low 32 bitsuint32_t datatype:6; // Type of the data pointed touint32_t access:2; // Access mode of the data pointed touint32_t padding:24; // Any use for these bits?};} entry_union;typedef union {uint32_t value;struct {uint32_t subindex:8;uint32_t index:16;uint32_t zero:8;};} key_union;static int32_t frag_find_index(const fragment *const frag, uint16_t target) {const uint_fast16_t last = frag->last;const uint16_t *index = frag->index;if (index == target)return 0;elseif (index > target || last < 1 || index < target)return -1;elseif (index == target)return last;int32_t i_min = 0;int32_t i_max = last;while (1) {int32_t i = (i_min + i_max) / 2;if (i == i_min)return -1;const uint16_t i_index = index;if (i_index < target)i_min = i;elseif (i_index > target)i_max = i;elsereturn i;}}// Preferably, this evaluates to all zeros.#define NO_ENTRY_FOUND (((entry_union){ .data = NULL, .datatype = DATATYPE_VOID, .access = ACCESS_CONST }).value)// Return value is actually entry_union, but because of ARMv6-m function call ABI, we need to use uint64_t instead (to pass this in registers).uint64_t find_entry(const fragment *frag, key_union key) {if (frag) {for (; frag->index != NULL; frag++) {int32_t i = frag_find_index(frag, key.index);if (i >= 0) {switch (frag->fragtype) {case FRAGTYPE_VAR:if (key.subindex)return NO_ENTRY_FOUND;return ((entry_union){.data = frag->var_values + i,.datatype = frag->var_acctypes.datatype,.access = frag->var_acctypes.access,}).value;case FRAGTYPE_REC:if (key.subindex >= frag->rec_offlens.length)return NO_ENTRY_FOUND;// Update i to point to the correct offset in rec_values and rec_acctypes arrays.i = frag->rec_offlens.offset + key.subindex;return ((entry_union){.data = frag->rec_values + i,.datatype = frag->rec_acctypes.datatype,.access = frag->rec_acctypes.access,}).value;default:return NO_ENTRY_FOUND;}}}}// Not located in any of the fragments.return NO_ENTRY_FOUND;}To implement the communications profile, we describe the VARs and the STRUCTs/ARRAYs in it:Code:const uint16_t comms_profile_var_index[] = {0x1000, 0x1001, 0x1002, 0x1005, 0x1006, 0x1007, 0x1012, 0x1013,};const acctype comms_profile_var_acctypes[] = {/* 0x1000 */ { .access = ACCESS_RO, .datatype = DATATYPE_U32 }, // Device type/* 0x1001 */ { .access = ACCESS_RO, .datatype = DATATYPE_U8 }, // Error register/* 0x1002 */ { .access = ACCESS_RO, .datatype = DATATYPE_U32 }, // Manufacturer status register/* 0x1005 */ { .access = ACCESS_RW, .datatype = DATATYPE_U32 }, // COB-ID SYNC/* 0x1006 */ { .access = ACCESS_RW, .datatype = DATATYPE_U32 }, // Communication cycle period/* 0x1007 */ { .access = ACCESS_RW, .datatype = DATATYPE_U32 }, // Synchronous window length/* 0x1012 */ { .access = ACCESS_RW, .datatype = DATATYPE_U32 }, // COB-ID time stamp message/* 0x1013 */ { .access = ACCESS_RW, .datatype = DATATYPE_U32 }, // High resolution time stamp};value32 comms_profile_var_values[sizeof comms_profile_var_acctypes / sizeof comms_profile_var_acctypes];#define device_type (comms_profile_var_values.u32)#define error_register (comms_profile_var_values.u8)#define manufacturer_status (comms_profile_var_values.u32)#define cob_id_sync (comms_profile_var_values.u32)#define comms_cycle_period (comms_profile_var_values.u32)#define sync_window_length (comms_profile_var_values.u32)#define cob_id_timestamp (comms_profile_var_values.u32)#define highres_timestamp (comms_profile_var_values.u32)const uint16_t comms_profile_rec_index = {0x1003,};const offlen comms_profile_rec_offlens = {/* 0x1003 */(offlen){ .offset = 0, .length = 3 },};const acctype comms_profile_rec_acctypes = {/* 0x1003, length 3 */{ .access = ACCESS_RW, .datatype = DATATYPE_U8 },{ .access = ACCESS_RO, .datatype = DATATYPE_U32 },{ .access = ACCESS_RO, .datatype = DATATYPE_U32 },};value32 comms_profile_rec_values = {/* 0x1003, length 3 */(value32){ .u8 = 2 },(value32){ .u32 = 0 },(value32){ .u32 = 0 },};#define error_count (comms_profile_rec_values.u8)#define error_1 (comms_profile_rec_values.u32)#define error_2 (comms_profile_rec_values.u32)Note thatrec_offlensindicates the first array index, and the number of elements, in therec_valuesandrec_acctypesarrays. Because the zeroth sub-index is also an element, the length is one more than in the CANopen Object Dictionary documentation, as there the sub-index 0 is not included in the count. It also means that stuff like writing to the error count (at sub-index 0) can be checked to not overflow the number of elements allocated.Next, we define a dictionary containing the above two fragments:Code:fragment dict[] = {{.index = comms_profile_var_index,.last = (sizeof comms_profile_var_index / sizeof comms_profile_var_index) - 1,.fragtype = FRAGTYPE_VAR,.var_acctypes = comms_profile_var_acctypes,.var_values = comms_profile_var_values,}, {.index = comms_profile_rec_index,.last = (sizeof comms_profile_rec_index / sizeof comms_profile_rec_index) - 1,.fragtype = FRAGTYPE_REC,.rec_offlens = comms_profile_rec_offlens,.rec_acctypes = comms_profile_rec_acctypes,.rec_values = comms_profile_rec_values,}, {// End of fragments.index = NULL,.last = 0,.fragtype = 0}};The order of the fragments doesn't matter much, although if the index ranges overlap as they do here, you might want to have the more often accessed ones first, to speed up the lookup a tiny, tiny bit.To locate an entry indict, you doconst entry_union entry = { .value = find_entry(dict, (key_union){ .index =index, .subindex =subindex}) };and ifentry.datais not NULL, it will point to where the value at that index and subindex is stored.entry.accesswill contain the access mode for it (ACCESS_CONST,ACCESS_RO,ACCESS_WO,ACCESS_RW), andentry.datatypetheDATATYPE_constant corresponding to thevalue32union type the data has.I amnotconvinced thefind_entry()interface is the best in practice, because what is best depends on how the object dictionary is used. I suspect you'll implement several versions of it, for different message types and use cases. However, written this way, it should be obvious how to extend/reimplement it as you wish.As is, it can already support VISIBLE_STRING and other types, if they are stored in thevar_valuesorrec_valuesarrays as pointers. The reason forvisible_string/const_visible_stringseparation in theDATATYPE_macros and members ofvalue32union, is to highlight the possibility of separating data values stored in Flash, and those stored in RAM, via the type, and hopefully help the compiler help us avoid trying to write to Flash memory.I did not implement sub-index 0xFF, although it can be implemented with not that much effort. It is optional, after all.If you want to export the DEFTYPE entries in your object dictionary, I recommend exporting a third fragment type, with just the 8-bit size of each type (0 for variable-length types like strings), exposed as a read-only UNSIGNED32 in sub-index 0. These are in separate index ranges, so would not slow down variable/array/record access at all, if you stick such fragments at the end of the fragment array.When looking at thecode GCC 9.2.1 generates for ARMv6-m/Cortex-M0+(godbolt.org), I kinda like it: it isn't horrible at all, in my opinion.

author: Simon
author_rank: Unknown
timestamp: February 02, 2025, 04:36:38 pm
content: I've been looking at C++ as I wanted to get into PC programming. I have just come across the "map container". This seems ideal for organizing the indices.

author: pardo-bsso
author_rank: Frequent Contributor
timestamp: June 16, 2025, 12:54:06 pm
content: Small advice as I had to create a CANOpen library at work (for a lot of weird reasons...)I'm sure that by now you already know this but if you value your time and sanity (if any is left) take an existing library.This is a good one to start:https://github.com/CANopenNode/CANopenNode

author: Simon
author_rank: Unknown
timestamp: June 16, 2025, 09:01:11 pm
content: I looked at it but as I am not a computer programmer I found it confusing and I am not clear on the RAM requirements. I typically work with 16kB. I am just starting to get interested in the PIC32CX with 256kB of RAM but that is some time off, as I have to stomach MPLABX because Microchip refuse to update Microchip studio that is far easier to work with and create a demo board to work on.To be honest CAN Open is not too difficult to work with and code for if you know what you are doing. I have started coding in C++, well static use of classes that helps me organize things. I have not yet got to the bottom of how to handle an object library as most of my considerations are about avoiding strategies that use up too much RAM. I want in the future to move to 256kB of RAM this will be easier later. What I have found is that I can create a lot of my library setup as const variables so that will limit the amount of data in memory to just the actual values.But yea that is time consuming so at least being able to read an electronic datasheet into some C/C++ code that I can use would make life much simpler.