Package: hexrec
docstestspackageLibrary to handle hexadecimal record filesFree software: BSD 2-Clause LicenseIntroductionThe purpose of this library is to provide simple but useful methods to load,
edit, and save hexadecimal record files.In the field of embedded systems, hexadecimal record files are the most common
way to share binary data to be written to the target non-volatile memory, such
as a EEPROM or microcontroller code flash.
Such binary data can contain compiled executable code, configuration data,
volatile memory dumps, etc.The most common file formats for hexadecimal record files areIntel HEX(.hex) andMotorola S-record(.srec).
Other common formats for binary data exchange for embedded systems include theExecutable and Linkable Format(.elf), hex dumps (byhexdumporxxd),
and raw binary files (.bin).A good thing about hexadecimal record files is that they are almostde-facto,
so every time a supplier has to give away its binary data it is either in HEX
or SREC, although ELF is arguably the most common for debuggable executables.A bad thing is that their support in embedded software toolsets is sometimes
flawed or only one of the formats is supported, while the supplier provides its
binary data in the other format.Another feature is that binary data is split into text record lines (thus their
name) protected by some kind of checksum. This is good for data exchange and
line-by-line writing to the target memory (in the old days), but this makes
in-place editing by humans rather tedious as data should be split, and the
checksum and other metadata have to be updated.All of the above led to the development of this library, which allows to,
for example:convert between hexadecimal record formats;merge/patch multiple hexadecimal record files of different formats;access every single record of a hexadecimal record file;build records through handy methods;edit sparse data in a virtual memory behaving like abytearray;extract or update only some parts of the binary data.DocumentationFor the full documentation, please refer to:https://hexrec.readthedocs.io/ArchitectureWithin thehexrecpackage itself are the symbols of the most commonly used
classes and functions.As the core of this library are record files, thehexrec.recordsis the
first module a user should look up.
It provides high-level functions to deal with record files, as well as classes
holding record data.Thehexrec.recordsallows to loadbytesparsevirtual memories, which
are as easy to use as the nativebytearray, but with sparse data blocks.Thehexrec.utilsmodule provides some miscellaneous utility stuff.hexrec.xxdis an emulation of thexxdcommand line utility delivered
byvim.The package can also be run as a command line tool, by running thehexrecpackage itself (python-mhexrec), providing some record file  utilities.
You can also create your own standalone executable, or download a precompiled
one from thepyinstallerfolder.The codebase is written in a simple fashion, to be easily readable and
maintainable, following some naive pythonicK.I.S.S.approach by choice.ExamplesTo have a glimpse of the features provided by this library, some simple but
common examples are shown in the following.Convert formatIt happens that some software tool only supports some hexadecimal record file
formats, or the format given to you is not handled properly, or simply you
prefer a format against another (e.g.SREC haslinearaddressing, while HEX
is in asegment:offsetfashion).In this example, a HEX file is converted to SREC.fromhexrecimportconvert_fileconvert_file('data.hex','data.srec')This can also be done by runninghexrecas a command line tool:$hexrecconvertdata.hexdata.srecAlternatively, by executing the package itself:$python-mhexrecconvertdata.hexdata.srecMerge filesIt is very common that the board factory prefers to receive a single file to
program the microcontroller, because a single file is simpler to manage for
them, and might be faster for their workers or machine, where every second
counts.This example shows how to merge a bootloader, an executable, and some
configuration data into a single file, in the order they are listed.fromhexrecimportmerge_filesinput_files=['bootloader.hex','executable.mot','configuration.s19']merge_files(input_files,'merged.srec')This can also be done by running thehexrecpackage as a command line tool:$hexrecmergebootloader.hexexecutable.motconfiguration.s19merged.srecAlternatively, these files can be merged manually viavirtual memory:fromhexrecimportload_memory,save_memoryfrombytesparseimportbytesparseinput_files=['bootloader.hex','executable.mot','configuration.s19']input_memories=[load_memory(filename)forfilenameininput_files]merged_memory=bytesparse()forinput_memoryininput_memories:merged_memory.write(0,input_memory)save_memory('merged.srec',merged_memory)Dataset generatorLet us suppose we are early in the development of the embedded system and we
need to test the current executable with some data stored in EEPROM.
We lack the software tool to generate such data, and even worse we need to test
100 configurations.
For the sake of simplicity, the data structure consists of 4096 random values
(0 to 1) offloattype, stored in little-endian at the address0xDA7A0000.importstruct,randomfromhexrecimportsave_chunkforindexinrange(100):values=[random.random()for_inrange(4096)]data=struct.pack('<4096f',*values)save_chunk(f'dataset_{index:02d}.mot',data,0xDA7A0000)Write a CRCUsually, the executable or the configuration data of an embedded system are
protected by a CRC, so that their integrity can be self-checked.Let us suppose that for some reason the compiler does not calculate such CRC
the expected way, and we prefer to do it with a script.This example shows how to load a HEX file, compute a CRC32 from the address0x1000to0x3FFB(0x3FFCexclusive), and write the calculated CRC
to0x3FFCin big-endian as a SREC file.
The rest of the data is left untouched.importbinascii,structfromhexrecimportsave_memorymemory=load_memory('data.srec')crc=binascii.crc32(memory[0x1000:0x3FFC])&0xFFFFFFFF# remove signmemory.write(0x3FFC,struct.pack('>L',crc))save_memory('data_crc.srec',memory)Trim for bootloaderWhen using a bootloader, it is very important that the application being
written does not overlap with the bootloader.  Sometimes the compiler still
generates stuff like a default interrupt table which should reside in the
bootloader, and we need to get rid of it, as well as everything outside the
address range allocated for the application itself.This example shows how to trim the application executable record file to the
allocated address range0x8000-0x1FFFF.  Being written to a flash
memory, unused memory byte cells default to0xFF.fromhexrecimportsave_chunkmemory=load_memory('app_original.hex')data=memory[0x8000:0x20000:b'\xFF']save_chunk('app_trimmed.srec',data,0x8000)This can also be done by running thehexrecpackage as a command line tool:$hexreccut-s0x8000-e0x20000-v0xFFapp_original.hexapp_trimmed.srecBy contrast, we need to fill the application range within the bootloader image
with0xFF, so that no existing application will be available again.
Also, we need to preserve the address range0x3F800-0x3FFFFbecause it
already contains some important data.fromhexrecimportload_memory,save_memorymemory=load_memory('boot_original.hex')memory.fill(0x8000,0x20000,b'\xFF')memory.clear(0x3F800,0x40000)save_memory('boot_fixed.srec',memory)With the command line interface, it can be done via a two-pass processing,
first to fill the application range, then to clear the reserved range.
Please note that the first command is chained to the second one via standard
output/input buffering (the virtual-file path, inintelformat as
perboot_original.hex).$hexrecfill-s0x8000-e0x20000-v0xFFboot_original.hex-|\hexrecclear-s0x3F800-e0x40000-iintel-boot_fixed.srec(newline continuation is backslash\for aUnix-likeshell, caret^for aDOSprompt).Export ELF physical programThe following example shows how to exportphysical programstored within anExecutable and Linkable File(ELF), compiled for a microcontroller.
As per the previous example, only data within the range0x8000-0x1FFFFare kept, with the rest of the memory filled with the0xFFvalue.fromhexrecimportsave_memoryfrombytesparseimportbytesparsefromelftools.elf.elffileimportELFFilewithopen('app.elf','rb')aself_stream:elf_file=ELFFile(elf_stream)memory=bytesparse(start=0x8000,endex=0x20000)# bounds setmemory.fill(pattern=b'\xFF')# between boundsforsegmentinelf_file.iter_segments(type='PT_LOAD'):addr=segment.header.p_paddrdata=segment.data()memory.write(addr,data)save_memory('app.srec',memory)InstallationFrom PyPI (might not be the latest version found ongithub):$pipinstallhexrecFrom the source code root directory:$pipinstall.DevelopmentTo run the all the tests:$pipinstalltox$tox